From b11b726c327dc204447b72be46d96df44bac74a8 Mon Sep 17 00:00:00 2001 From: Bianca Danforth Date: Mon, 8 Oct 2018 21:24:48 -0700 Subject: [PATCH] Incorporate feedback from various sources METRICS.md * Reorganized the questions in the Analysis section to fall under broad categories of questions. * Added a scalar 'supported_sites' to keep track of the number of times the user visits one of the five supported websites. * Added an 'extra_keys' section that explains up front what each key represents. * Made data types for all scalar and event telemetry values more explicit. * Added a 'detect_price_change' event. * Updated and added new keys to provide more context around certain events and ensure data collection is Category 1 or 2 at most. telemetry.js * Updated registered events and scalars to match METRICS.md Note: While this patch does not record any events or scalars itself, recording can be tested with the following after 'telemetry.recordProbes()' in './src/background/index.js': ```javascript await telemetry.scalarAdd('supported_sites', 1); await telemetry.recordEvent( 'badge_toolbar_button', 'toolbar_button', null, { badge_type: 'add', tracked_prods: '5', }, ); ``` --- docs/METRICS.md | 289 +++++++++++++++++++++++++----------- src/background/index.js | 2 +- src/background/telemetry.js | 145 ++++++++++++------ 3 files changed, 306 insertions(+), 130 deletions(-) diff --git a/docs/METRICS.md b/docs/METRICS.md index 87a1f92..ab47dca 100644 --- a/docs/METRICS.md +++ b/docs/METRICS.md @@ -11,72 +11,92 @@ A summary of the metrics the Price Alerts extension will record. * **Fathom**: A [JavaScript framework](https://github.com/erikrose/fathom) used to extract product information from a Product Page. * **Product Page**: A webpage ([example](https://www.amazon.com/LEGO-Fantastic-Beasts-Grindelwald-Grindelwalds/dp/B07BKQXCZR/ref=sr_1_3_sspa?s=toys-and-games&ie=UTF8&qid=1538418041&sr=1-3-spons&keywords=legos&psc=1)) displaying a single product that the user could purchase online. * **Extraction**: The process of extracting the necessary product information from a Product Page using one of two methods: Fathom or fallback (CSS selectors or Open Graph attributes). -* **Product Card**: A list item in the list of tracked products for which the user has opted to receive Price Alerts displayed on the browserAction popup. Each Product Card displays the product title, image and price. +* **Product Card**: A product list item in the list of tracked products for which the user has opted to receive Price Alerts displayed on the browserAction popup. Each Product Card displays the product title, image and price among other information. * **Survey**: a short survey collecting user feedback. * **Onboarding Popup**: The popup displayed when the user has zero products tracked, including the first time the popup is opened. +* **Supported Sites**: For the initial launch (a.k.a. MVP, Minimum Viable Product) of this extension, we are limiting the sites supported by this feature to [five websites](https://github.com/mozilla/webext-commerce/issues/36#issuecomment-409641491): Amazon, Ebay, Walmart, Home Depot and Best Buy. ## Analysis Data collected by the Price Alerts extension will be used to answer the following questions: -### User Engagement +(Note: For any questions related to general user shopping behavior, the data about what sites users visit is limited to the Supported Sites for the MVP.) -- Is the Price Alerts experience compelling? - - How often do users set price alerts? - - Do alerts cause users to re-engage with our shopping feature? - - How often do they respond to price alerts? - - At what threshold do users respond to price changes? - - How do users respond when they receive a price alert? - - Do they click through to the product page? - - In what intervals? - - In what circumstances? - - How often are UI elements used? - - What are users' tolerance for inaccuracy? - - Are alerts annoying to users? - - How often do users keep alerts? - - How often do users delete alerts? - - How much time passes before the "set alert" button is available in the drop down? -- Where do users use Price Alerts? - - For what kinds of products? - - Which sites do users track products on most? - - What's the price distribution of products users track? - - -### Experiment Health - -- Can users be compelled to report positive experiences as well as negative ones? +#### Does the extension work? +- How often do product prices change, irrespective of user action? - How often are products detected on supported product pages? (#125) - How often is Fathom recognizing products on a page? (#125) - On which sites do users report the most problems? - How often do product prices change, irrespective of user action? - How many price alerts are received? - - In what intervals? -### Performance +#### Do people use it? +- At what threshold do users respond to price changes? +- How many products are users tracking? + - Does number of tracked items track to feature engagement? +- Are alerts annoying to users? +- Can users be compelled to report positive experiences as well as negative ones? -There are open questions about whether and how much Fathom may affect page performance. Time permitting, we may ask: -- What is the frame rate for a page while Fathom is running? (#124) +#### What parts of it do they use? +- Do users set price alerts? +- How do users respond to price alerts? +- Do they delete alerts? + +#### How much do they use it? +- How much of browsing is shopping? +- How often do users shop (daily, weekly, monthly, etc.)? +- How often do users set price alerts? +- How often do users respond to price alerts? +- Do alerts cause users to re-engage with our shopping feature? +- At what price changes do users re-engage with saved products? +- How often do users keep alerts? +- How often do they delete them? +- What are users' tolerance for inaccuracy? + + +#### In what circumstances do they use it? +- Which sites do users track products on most? +- What sites do users shop on the most? +- Which sites do users browse on the most? +- What's the price distribution of products users track? + + +#### Does it affect user behavior in Firefox? +- Does it increase shopping browsing? +- Does it increase shopping purchases? +- Did it impact search revenue? (Standard telemetry) +- Did it impact retention? (Standard telemetry) +- Did it impact satisfaction? (Survey) +- Did it impact usage of Firefox? (Standard telemetry) ### Questions to answer in future experiments -Answering these questions requires controlled A/B testing or more extensive data collection. +While some of these questions may be partially answerable by this extension, answering them thorougly requires controlled A/B testing and/or more extensive data collection on more sites. -- Does our feature increase shopping browsing? -- Does our feature increase shopping purchase? +- How much of browsing is shopping? +- How often do users shop (daily, weekly, monthly, etc.)? +- Does it increase shopping browsing? +- Does it increase shopping purchases? - Which sites do users browse on the most? - On what sites do users shop? ## Sample Pings -We will be sending pings using [Event Telemetry](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html) via the [WebExtensions Telemetry API](https://bugzilla.mozilla.org/show_bug.cgi?id=1280234). +We will be sending pings using [Scalar](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/scalars.html) and [Event Telemetry](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html) via the [WebExtensions Telemetry API](https://bugzilla.mozilla.org/show_bug.cgi?id=1280234). -Each event will exist as part of the `main` ping, under `payload.processes.dynamic.events` as an array. Below is a sample ping for the `badge_toolbar_button` event: +Each scalar will exist as part of the `main` ping under `payload.processes.dynamic.scalars` as a key-value pair in the `scalars` object. The data types for values are one of the scalar types (number, string or boolean). -```json +Each event will exist as part of the `main` ping under `payload.processes.dynamic.events` as an array in the `events` array. The data types of individual fields for each event follow the Event Telemetry [serialization format](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html#serialization-format). + +The telemetry category for both scalars and events is `'extension.pricealerts'`. + +Below is a sample ping for the `supported_sites` scalar and `badge_toolbar_button` event. + +```javascript { "type": "main", // ... @@ -85,29 +105,62 @@ Each event will exist as part of the `main` ping, under `payload.processes.dynam "processes": { // ... "dynamic": { - // ... + "scalars": { + "extension.pricealerts.supported_sites": 5 + }, "events": [ [ - 618573, - "extension.price_alerts", + 1634, + "extension.pricealerts", "badge_toolbar_button", "toolbar_button", null, { - "badge_type": "add" + "badge_type": "add", + "tracked_prods": "5" } ] ] } + // ... } + // ... } + // ... } ``` -## Collection (User Events) +## `extra_keys` -Each event here is the value of the `methods` key used in Event Telemetry. +**Note: All `extra_keys` and their values are strings.** + +- `'badge_type'`: Indicates what, if any, badge was present on the browserAction toolbar button. One of 'add', 'price_alert', or 'none'. +- `'extraction_id'`: A unique identifier to associate an extraction attempt to an extraction completion event for a given page. +- `'is_bg_update'`: 'true' if the extraction is associated with a background price check; otherwise 'false'. +- `method`: The extraction method that was successful, if any. One of: 'fathom', 'fallback' or 'neither'. A value of 'neither' means that extraction failed. +- `'price'`: The price of the product in subunits (e.g. a $10.00 product would have a value of `'1000'`). +- `'price_alert'`: 'true' if the product has an active price alert; otherwise 'false'. +- `'price_orig'`: The original price of the product in subunits (e.g. a $10.00 product would have a value of `'1000'`). +- `'product_index'`: The index of the product in the product listing. The top most list item is index '0'. The list is sorted by date added, descending (#113). +- `'product_key'`: A unique identifier for the product relative to other products for a given user. This key is _not_ unique to the product across all users. +- `'tracked_prods'`: The number of products the user is tracking. + +## Collection (Scalars) + +### `supported_sites` + +Increments when the user navigates to a Supported Site. + +#### Payload properties +- `scalarName`: String + - `'supported_sites'` +- `kind`: String + - `'count'` +- `record_on_release`: Boolean + - `true` + +## Collection (User Events) ### `open_popup` @@ -116,33 +169,41 @@ Fired when the user clicks the Price Alerts browserAction toolbar button to open #### Payload properties -- `objects`: +- `methods`: String + - `'open_popup'` +- `objects`: String - `'toolbar_button'` -- `extra_keys`: - - `'badge-type'`: String indicating what, if any, badge was present on the toolbar. One of 'add', 'price_alert', or 'none'. +- `extra_keys`: Object + - `'badge-type'` + - `'tracked_prods'` ### `open_external_page` -Fired when the user clicks on a UI element that opens a page in a new tab. +Fired when the user clicks on a UI element that opens a page in a new tab. All links are present only in the Onboarding Popup. #### Payload properties -- `objects`: (All links are in the Onboarding Popup only.) - - `'help_button'`: Sends the user to a Price Alerts support.mozilla.org page. - - `'feedback_button'`: Sends the user to a feedback Survey. - - `'system_notification'`: Sends the user to the product page for the price alert displayed in the notification. - - `'product_card'`: Sends the user to the product page for the given Product Card. +- `methods`: String + - `'open_external_page'` +- `objects`: String; one of... - `'amazon_link'`: Sends the user to Amazon. - `'best_buy_link'`: Sends the user to Best Buy. - `'ebay_link'`: Sends the user to Ebay. + - `'feedback_button'`: Sends the user to a feedback Survey. + - `'help_button'`: Sends the user to a Price Alerts support.mozilla.org page. - `'home_depot_link'`: Sends the user to Home Depot. - - `'walmart_link'`: Sends the user to Walmart. - `'learn_more_link'`: Sends the user to a Price Alerts support.mozilla.org page. -- `extra_keys`: For objects of type `'system_notification'` and `'product'` only - - `'product_id'`: String. The unique product identifier. - - `'price'`: Number. The price of the product in subunits (e.g. a $10.00 product would have a value of `1000`). - - `'price_original'`: Number. The original price of the product in subunits (e.g. a $10.00 product would have a value of `1000`). + - `'product_card'`: Sends the user to the product page for the given Product Card. + - `'system_notification'`: Sends the user to the product page for the price alert displayed in the notification. + - `'walmart_link'`: Sends the user to Walmart. +- `extra_keys`: Object; for objects of type 'system_notification' and 'product' only ('tracked_prods' excepted) + - `'price'` + - `'price_alert'` + - `'price_orig'` + - `'product_index'` + - `'product_key'` + - `'tracked_prods'` ### `add_product` @@ -152,10 +213,14 @@ Fired when the user clicks the add product button in the browserAction popup. #### Payload properties -- `objects`: +- `methods`: String + - `'add_product'` +- `objects`: String - `'add_button'` -- `extra_keys`: - - `'product_id'`: String. The unique product identifier. +- `extra_keys`: Object + - `'price'` + - `'product_key'` + - `'tracked_prods'` ### `delete_product` @@ -165,10 +230,17 @@ Fired when the user clicks a delete product button in a Product Card in the brow #### Payload properties -- `objects`: +- `methods`: String + - `'delete_product'` +- `objects`: String - `'delete_button'` -- `extra_keys`: - - `'product_id'`: String. The unique product identifier. +- `extra_keys`: Object + - `'price'` + - `'price_alert'` + - `'price_orig'` + - `'product_index'` + - `'product_key'` + - `'tracked_prods'` ### `undo_delete_product` @@ -178,10 +250,17 @@ Fired when the user clicks an undo button in a Product Card in the browserAction #### Payload properties -- `objects`: +- `methods`: String + - `'undo_delete_product'` +- `objects`: String - `'undo_button'` -- `extra_keys`: - - `'product_id'`: String. The unique product identifier. +- `extra_keys`: Object + - `'price'` + - `'price_alert'` + - `'price_orig'` + - `'product_index'` + - `'product_key'` + - `'tracked_prods'` ### `uninstall` @@ -191,8 +270,12 @@ Fired when the user uninstalls the extension. #### Payload properties -- `objects`: +- `methods`: String - `'uninstall'` +- `objects`: String + - `'uninstall'` +- `extra_keys`: Object + - `'tracked_prods'` ### `hide_toolbar_button` @@ -202,13 +285,35 @@ Fired when the user hides the extension's browserAction toolbar button from the #### Payload properties -- `objects`: +- `methods`: String + - `'hide_toolbar_button'` +- `objects`: String - `'toolbar_button'` +- `extra_keys`: Object + - `'badge-type'` + - `'tracked_prods'` ## Collection (Non-User Events) +### `detect_price_change` + +Fired whenever a price change (of any magnitude and in any direction) is detected. + + +#### Payload properties + +- `methods`: String + - `'detect_price_change'` +- `objects`: String + - `'product_page'` +- `extra_keys`: Object + - `'price'` + - `'price_orig'` + - `'product_key'` + - `'tracked_prods'` + ### `badge_toolbar_button` Fired whenever the toolbar is badged in response to: @@ -218,25 +323,31 @@ Fired whenever the toolbar is badged in response to: #### Payload properties -- `objects`: +- `methods`: String + - `'badge_toolbar_button'` +- `objects`: String - `'toolbar_button'` -- `extra_keys`: - - `'badge-type'`: String indicating what, if any, badge was present on the toolbar. One of 'add' or 'price_alert'. +- `extra_keys`: Object + - `'badge-type'` + - `'tracked_prods'` -### `send_system_notice` +### `send_notification` Fired whenever a system notification is sent to the user notifying them of a price alert. #### Payload properties -- `objects`: +- `methods`: String + - `'send_notification'` +- `objects`: String - `'system_notification'` -- `extra_keys`: - - `'product_id'`: String. The unique product identifier. - - `'price'`: Number. The price of the product in subunits (e.g. a $10.00 product would have a value of `1000`). - - `'price_original'`: Number. The original price of the product in subunits (e.g. a $10.00 product would have a value of `1000`). +- `extra_keys`: Object + - `'price'` + - `'price_orig'` + - `'product_key'` + - `'tracked_prods'` ### `attempt_extraction` @@ -246,10 +357,14 @@ Fired whenever a supported page loads and the add-on attempts to extract product #### Payload properties -- `objects`: - - `'page'` -- `extra_keys`: - - `url`: String. The url of the page the extraction script is running in +- `methods`: String + - `'attempt_extraction'` +- `objects`: String + - `'product_page'` +- `extra_keys`: Object + - `'extraction_id'` + - `'is_bg_update'` + - `'tracked_prods'` ### `complete-extraction` @@ -259,11 +374,15 @@ Fired whenever extraction on a supported page completes, whether or not the extr #### Payload properties -- `objects`: - - `'page'` -- `extra_keys`: - - `url`: String. The url of the page the extraction script is running in - - `method`: String. The extraction method that was successful, if any. One of: 'fathom', 'fallback' or 'neither'. A value of 'neither' means that extraction failed. +- `methods`: String + - `'complete_extraction'` +- `objects`: String + - `'product_page'` +- `extra_keys`: Object + - `'extraction_id'` + - `'is_bg_update'` + - `method` + - `'tracked_prods'` ## Opt-out diff --git a/src/background/index.js b/src/background/index.js index e555164..d29008c 100644 --- a/src/background/index.js +++ b/src/background/index.js @@ -21,7 +21,7 @@ import {loadStateFromStorage} from 'commerce/state/sync'; import telemetry from 'commerce/background/telemetry'; (async function main() { - telemetry.init(); + telemetry.registerProbes(); // Set browser action default badge color, which can't be set via manifest browser.browserAction.setBadgeBackgroundColor({ diff --git a/src/background/telemetry.js b/src/background/telemetry.js index bb48833..0c09291 100644 --- a/src/background/telemetry.js +++ b/src/background/telemetry.js @@ -3,127 +3,184 @@ * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ /** - * TODO: add desc. + * Registers and records telemetry scalars and events throughout the extension. * @module */ const telemetry = { - EVENT_CATEGORY: 'extension.price_alerts', + // Scalar category names can't have underscores + CATEGORY: 'extension.pricealerts', + scalars: { + // Incremented when the user navigates to a supported site (e.g. Amazon) + supported_sites: { + kind: 'count', + record_on_release: true, + }, + }, events: { - /** - * User Events - */ + // User Events // User clicks toolbar button to open the popup open_popup: { methods: ['open_popup'], objects: ['toolbar_button'], - extra_keys: ['badge_type'], + extra_keys: [ + 'badge_type', + 'tracked_prods', + ], }, // User clicks on a UI element opening a page in a new tab open_external_page: { methods: ['open_external_page'], objects: [ - 'help_button', - 'feedback_button', - 'system_notification', - 'product_card', 'amazon_link', 'best_buy_link', 'ebay_link', + 'feedback_button', + 'help_button', 'home_depot_link', - 'walmart_link', 'learn_more_link', + 'product_card', + 'system_notification', + 'walmart_link', ], - // For objects of type 'system_notification' and 'product' only + // For objects of type 'system_notification' and 'product' only ('tracked_prods' excepted) extra_keys: [ - 'product_id', 'price', - 'price_original', + 'price_alert', + 'price_orig', + 'product_index', + 'product_key', + 'tracked_prods', ], }, // User adds a product to the product listing add_product: { methods: ['add_product'], objects: ['add_button'], - extra_keys: ['product_id'], + extra_keys: [ + 'price', + 'product_key', + 'tracked_prods', + ], }, // User deletes a product from the product listing delete_product: { methods: ['delete_product'], objects: ['delete_button'], - extra_keys: ['product_id'], + extra_keys: [ + 'price', + 'price_alert', + 'price_orig', + 'product_index', + 'product_key', + 'tracked_prods', + ], }, // User undeletes a product from the product listing undo_delete_product: { methods: ['undo_delete_product'], objects: ['undo_button'], - extra_keys: ['product_id'], + extra_keys: [ + 'price', + 'price_alert', + 'price_orig', + 'product_index', + 'product_key', + 'tracked_prods', + ], }, // User uninstalls the extension uninstall: { methods: ['uninstall'], objects: ['uninstall'], - extra_keys: [], + extra_keys: ['tracked_prods'], }, // User hides the toolbar button for the extension hide_toolbar_button: { methods: ['hide_toolbar_button'], objects: ['toolbar_button'], - extra_keys: [], + extra_keys: [ + 'badge_type', + 'tracked_prods', + ], }, - /** - * Non-user Events - */ + + // Non-User Events + // There is a price change on a tracked product + detect_price_change: { + methods: ['detect_price_change'], + objects: ['product_page'], + extra_keys: [ + 'price', + 'price_orig', + 'product_key', + 'tracked_prods', + ], + }, + // Toolbar button is badged either due to a price alert or an extracted product badge_toolbar_button: { methods: ['badge_toolbar_button'], objects: ['toolbar_button'], - extra_keys: ['badge_type'], + extra_keys: [ + 'badge_type', + 'tracked_prods', + ], }, // System notification is sent notifying user of a price alert - send_system_notice: { - methods: ['send_system_notice'], + send_notification: { + methods: ['send_notification'], objects: ['system_notification'], extra_keys: [ - 'product_id', - 'price_original', 'price', + 'price_orig', + 'product_key', + 'tracked_prods', ], }, // Product extraction is attempted on the content page attempt_extraction: { methods: ['attempt_extraction'], - objects: ['page'], - extra_keys: ['url'], + objects: ['product_page'], + extra_keys: [ + 'extraction_id', + 'is_bg_update', + 'tracked_prods', + ], }, // Product extraction is completed on the content page complete_extraction: { methods: ['complete_extraction'], - objects: ['page'], + objects: ['product_page'], extra_keys: [ - 'url', + 'extraction_id', + 'is_bg_update', 'method', + 'tracked_prods', ], }, }, - async init() { - await browser.telemetry.registerEvents(this.EVENT_CATEGORY, this.events); + async registerProbes() { + await browser.telemetry.registerScalars(this.CATEGORY, this.scalars); + await browser.telemetry.registerEvents(this.CATEGORY, this.events); }, async recordEvent(method, object, value, extra) { if (!browser.telemetry.canUpload()) { return; } - try { - await browser.telemetry.recordEvent( - this.EVENT_CATEGORY, - method, - object, - value, - extra, - ); - } catch (error) { - throw new Error(`recordEvent failed with error: ${error}`); - } + await browser.telemetry.recordEvent( + this.CATEGORY, + method, + object, + value, + extra, + ); + }, + async scalarAdd(scalarName, value) { + await browser.telemetry.scalarAdd( + `${this.CATEGORY}.${scalarName}`, + value, + ); }, };