Incorporate more feedback on the metrics used for the extension.

METRICS.md
* For the `open_external_page` probe, moved the ‘objects’ enums to their own ‘extra_key’, ‘element’. This is so that the schema doesn’t have to change if the content of these UI elements changes down the road (feedback from Osmose).
* Arrange list items in Definitions section in alphabetical order and add Price Alert definition.
* Move questions under ‘Does it affect user behavior in Firefox?’ in the Analysis section to the ‘Questions to answer in future experiments’ subsection.
* Fixed probe name typos.
* Ensured consistent whitespace and use of markdown symbols.

telemetry.js
* Instead of exporting a single object, export the methods from the module directly.
* Update probes based on changes to METRICS.md

config.js
* Unrelated to the telemetry issue here; fixed a typo I noticed.
This commit is contained in:
Bianca Danforth 2018-10-10 18:58:06 -07:00
Родитель b11b726c32
Коммит d70ffd5d0e
4 изменённых файлов: 254 добавлений и 283 удалений

Просмотреть файл

@ -8,20 +8,21 @@ A summary of the metrics the Price Alerts extension will record.
## Definitions
* **Fathom**: A [JavaScript framework](https://github.com/erikrose/fathom) used to extract product information from a Product Page.
* **Product Page**: A webpage ([example](https://www.amazon.com/LEGO-Fantastic-Beasts-Grindelwald-Grindelwalds/dp/B07BKQXCZR/ref=sr_1_3_sspa?s=toys-and-games&ie=UTF8&qid=1538418041&sr=1-3-spons&keywords=legos&psc=1)) displaying a single product that the user could purchase online.
* **Extraction**: The process of extracting the necessary product information from a Product Page using one of two methods: Fathom or fallback (CSS selectors or Open Graph attributes).
* **Fathom**: A [JavaScript framework](https://github.com/erikrose/fathom) used to extract product information from a Product Page.
* **Price Alert**: An alert that occurs when a tracked product's price _decreases_ below a certain absolute or percentage threshold. For the MVP, the default thresholds are specified in `./src/config.js`.
* **Product Card**: A product list item in the list of tracked products for which the user has opted to receive Price Alerts displayed on the browserAction popup. Each Product Card displays the product title, image and price among other information.
* **Product Page**: A webpage ([example](https://www.amazon.com/LEGO-Fantastic-Beasts-Grindelwald-Grindelwalds/dp/B07BKQXCZR/ref=sr_1_3_sspa?s=toys-and-games&ie=UTF8&qid=1538418041&sr=1-3-spons&keywords=legos&psc=1)) displaying a single product that the user could purchase online.
* **Supported Sites**: For the initial launch (a.k.a. MVP, Minimum Viable Product) of this extension, we are limiting the sites supported by this feature to [five websites](https://github.com/mozilla/webext-commerce/issues/36#issuecomment-409641491): Amazon, Ebay, Walmart, Home Depot and Best Buy.
* **Survey**: a short survey collecting user feedback.
* **Onboarding Popup**: The popup displayed when the user has zero products tracked, including the first time the popup is opened.
* **Supported Sites**: For the initial launch (a.k.a. MVP, Minimum Viable Product) of this extension, we are limiting the sites supported by this feature to [five websites](https://github.com/mozilla/webext-commerce/issues/36#issuecomment-409641491): Amazon, Ebay, Walmart, Home Depot and Best Buy.
## Analysis
Data collected by the Price Alerts extension will be used to answer the following questions:
(Note: For any questions related to general user shopping behavior, the data about what sites users visit is limited to the Supported Sites for the MVP.)
Note: For any questions related to general user shopping behavior, the data about what sites users visit is limited to the Supported Sites for the MVP.
#### Does the extension work?
- How often do product prices change, irrespective of user action?
@ -29,8 +30,7 @@ Data collected by the Price Alerts extension will be used to answer the followin
- How often is Fathom recognizing products on a page? (#125)
- On which sites do users report the most problems?
- How often do product prices change, irrespective of user action?
- How many price alerts are received?
- How many Price Alerts are received?
#### Do people use it?
- At what threshold do users respond to price changes?
@ -39,51 +39,47 @@ Data collected by the Price Alerts extension will be used to answer the followin
- Are alerts annoying to users?
- Can users be compelled to report positive experiences as well as negative ones?
#### What parts of it do they use?
- Do users set price alerts?
- How do users respond to price alerts?
- Do users set Price Alerts?
- How do users respond to Price Alerts?
- Do they delete alerts?
#### How much do they use it?
- How much of browsing is shopping?
- How often do users shop (daily, weekly, monthly, etc.)?
- How often do users set price alerts?
- How often do users respond to price alerts?
- How often do users set Price Alerts?
- How often do users respond to Price Alerts?
- Do alerts cause users to re-engage with our shopping feature?
- At what price changes do users re-engage with saved products?
- How often do users keep alerts?
- How often do they delete them?
- What are users' tolerance for inaccuracy?
#### In what circumstances do they use it?
- Which sites do users track products on most?
- What sites do users shop on the most?
- Which sites do users browse on the most?
- What's the price distribution of products users track?
#### Does it affect user behavior in Firefox?
- Does it increase shopping browsing?
- Does it increase shopping purchases?
- Did it impact search revenue? (Standard telemetry)
- Did it impact retention? (Standard telemetry)
- Did it impact satisfaction? (Survey)
- Did it impact usage of Firefox? (Standard telemetry)
### Questions to answer in future experiments
While some of these questions may be partially answerable by this extension, answering them thorougly requires controlled A/B testing and/or more extensive data collection on more sites.
- How much of browsing is shopping?
- How often do users shop (daily, weekly, monthly, etc.)?
#### Does it affect user behavior in Firefox?
- Does it increase shopping browsing?
- Does it increase shopping purchases?
- Did it impact search revenue?
- Did it impact retention?
- Did it impact satisfaction?
- Did it impact usage of Firefox?
#### How well do we understand aggregate user shopping behavior?
- How much of browsing is shopping?
- How often do users shop (daily, weekly, monthly, etc.)?
- Which sites do users browse on the most?
- On what sites do users shop?
## Sample Pings
We will be sending pings using [Scalar](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/scalars.html) and [Event Telemetry](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html) via the [WebExtensions Telemetry API](https://bugzilla.mozilla.org/show_bug.cgi?id=1280234).
@ -92,7 +88,7 @@ Each scalar will exist as part of the `main` ping under `payload.processes.dynam
Each event will exist as part of the `main` ping under `payload.processes.dynamic.events` as an array in the `events` array. The data types of individual fields for each event follow the Event Telemetry [serialization format](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html#serialization-format).
The telemetry category for both scalars and events is `'extension.pricealerts'`.
The telemetry category for both scalars and events is `'extension.pricealerts'`. Note: Scalar telemetry category names [do not allow underscores](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/scalars.html#scalar-limitations).
Below is a sample ping for the `supported_sites` scalar and `badge_toolbar_button` event.
@ -133,60 +129,19 @@ Below is a sample ping for the `supported_sites` scalar and `badge_toolbar_butto
## `extra_keys`
**Note: All `extra_keys` and their values are strings.**
`extra_keys` are keys in the [optional `extra` object field](https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/events.html#serialization-format) for telemetry events. All `extra_keys` and their values are strings.
- `'badge_type'`: Indicates what, if any, badge was present on the browserAction toolbar button. One of 'add', 'price_alert', or 'none'.
- `'extraction_id'`: A unique identifier to associate an extraction attempt to an extraction completion event for a given page.
- `'is_bg_update'`: 'true' if the extraction is associated with a background price check; otherwise 'false'.
- `method`: The extraction method that was successful, if any. One of: 'fathom', 'fallback' or 'neither'. A value of 'neither' means that extraction failed.
- `'price'`: The price of the product in subunits (e.g. a $10.00 product would have a value of `'1000'`).
- `'price_alert'`: 'true' if the product has an active price alert; otherwise 'false'.
- `'price_alert'`: 'true' if the product has an active Price Alert; otherwise 'false'.
- `'price_orig'`: The original price of the product in subunits (e.g. a $10.00 product would have a value of `'1000'`).
- `'product_index'`: The index of the product in the product listing. The top most list item is index '0'. The list is sorted by date added, descending (#113).
- `'product_key'`: A unique identifier for the product relative to other products for a given user. This key is _not_ unique to the product across all users.
- `'tracked_prods'`: The number of products the user is tracking.
## Collection (Scalars)
### `supported_sites`
Increments when the user navigates to a Supported Site.
#### Payload properties
- `scalarName`: String
- `'supported_sites'`
- `kind`: String
- `'count'`
- `record_on_release`: Boolean
- `true`
## Collection (User Events)
### `open_popup`
Fired when the user clicks the Price Alerts browserAction toolbar button to open the popup.
#### Payload properties
- `methods`: String
- `'open_popup'`
- `objects`: String
- `'toolbar_button'`
- `extra_keys`: Object
- `'badge-type'`
- `'tracked_prods'`
### `open_external_page`
Fired when the user clicks on a UI element that opens a page in a new tab. All links are present only in the Onboarding Popup.
#### Payload properties
- `methods`: String
- `'open_external_page'`
- `objects`: String; one of...
- `'element'`: The extension UI element that the user clicked to open a page in a new tab. Note: All `*_link` elements exist in the Onboarding Popup only. One of...
- `'amazon_link'`: Sends the user to Amazon.
- `'best_buy_link'`: Sends the user to Best Buy.
- `'ebay_link'`: Sends the user to Ebay.
@ -195,22 +150,60 @@ Fired when the user clicks on a UI element that opens a page in a new tab. All l
- `'home_depot_link'`: Sends the user to Home Depot.
- `'learn_more_link'`: Sends the user to a Price Alerts support.mozilla.org page.
- `'product_card'`: Sends the user to the product page for the given Product Card.
- `'system_notification'`: Sends the user to the product page for the price alert displayed in the notification.
- `'system_notification'`: Sends the user to the product page for the Price Alert displayed in the notification.
- `'walmart_link'`: Sends the user to Walmart.
- `extra_keys`: Object; for objects of type 'system_notification' and 'product' only ('tracked_prods' excepted)
- `'price'`
- `'price_alert'`
- `'price_orig'`
- `'product_index'`
- `'product_key'`
## Collection (Scalars)
### `supported_sites`
Increments when the user navigates to a Supported Site.
#### Payload properties
Note: The key below is the scalar name prepended with the telemetry category, separated by a `'.'`.
- `'extension.pricealerts.supported_sites'`: Number; the current count for the number of times the user has visited a Supported Site during the session.
## Collection (User Events)
### `open_popup`
Fired when the user clicks the Price Alerts browserAction toolbar button to open the popup.
#### Payload properties
- `methods`: String
- `'open_popup'`
- `objects`: String
- `'toolbar_button'`
- `extra_keys`: Object
- `'badge_type'`
- `'tracked_prods'`
### `open_external_page`
Fired when the user clicks on a UI element in the extension that opens a page in a new tab.
#### Payload properties
- `methods`: String
- `'open_external_page'`
- `objects`: String
- `'ui_element'`
- `extra_keys`: Object; which keys are included depends on the value of the `'element'` extra key.
- All values:
- `'tracked_prods'`
- `'element'`
- `'system_notification'` and `'product_card'` only:
- `'price'`
- `'price_alert'`
- `'price_orig'`
- `'product_key'`
- `'product_card'` only:
- `'product_index'`
### `add_product`
Fired when the user clicks the add product button in the browserAction popup.
#### Payload properties
- `methods`: String
@ -222,12 +215,10 @@ Fired when the user clicks the add product button in the browserAction popup.
- `'product_key'`
- `'tracked_prods'`
### `delete_product`
Fired when the user clicks a delete product button in a Product Card in the browserAction popup.
#### Payload properties
- `methods`: String
@ -242,12 +233,10 @@ Fired when the user clicks a delete product button in a Product Card in the brow
- `'product_key'`
- `'tracked_prods'`
### `undo_delete_product`
Fired when the user clicks an undo button in a Product Card in the browserAction popup.
#### Payload properties
- `methods`: String
@ -262,12 +251,10 @@ Fired when the user clicks an undo button in a Product Card in the browserAction
- `'product_key'`
- `'tracked_prods'`
### `uninstall`
Fired when the user uninstalls the extension.
#### Payload properties
- `methods`: String
@ -277,12 +264,10 @@ Fired when the user uninstalls the extension.
- `extra_keys`: Object
- `'tracked_prods'`
### `hide_toolbar_button`
Fired when the user hides the extension's browserAction toolbar button from the browser chrome.
#### Payload properties
- `methods`: String
@ -290,18 +275,16 @@ Fired when the user hides the extension's browserAction toolbar button from the
- `objects`: String
- `'toolbar_button'`
- `extra_keys`: Object
- `'badge-type'`
- `'badge_type'`
- `'tracked_prods'`
## Collection (Non-User Events)
### `detect_price_change`
Fired whenever a price change (of any magnitude and in any direction) is detected.
#### Payload properties
- `methods`: String
@ -318,8 +301,7 @@ Fired whenever a price change (of any magnitude and in any direction) is detecte
Fired whenever the toolbar is badged in response to:
1. A successful product extraction or
2. A price alert
2. A Price Alert
#### Payload properties
@ -328,14 +310,12 @@ Fired whenever the toolbar is badged in response to:
- `objects`: String
- `'toolbar_button'`
- `extra_keys`: Object
- `'badge-type'`
- `'badge_type'`
- `'tracked_prods'`
### `send_notification`
Fired whenever a system notification is sent to the user notifying them of a price alert.
Fired whenever a system notification is sent to the user notifying them of a Price Alert.
#### Payload properties
@ -349,12 +329,10 @@ Fired whenever a system notification is sent to the user notifying them of a pri
- `'product_key'`
- `'tracked_prods'`
### `attempt_extraction`
Fired whenever a supported page loads and the add-on attempts to extract product information from the page.
#### Payload properties
- `methods`: String
@ -366,12 +344,10 @@ Fired whenever a supported page loads and the add-on attempts to extract product
- `'is_bg_update'`
- `'tracked_prods'`
### `complete-extraction`
### `complete_extraction`
Fired whenever extraction on a supported page completes, whether or not the extraction was successful.
#### Payload properties
- `methods`: String
@ -390,10 +366,10 @@ Fired whenever extraction on a supported page completes, whether or not the extr
All data collection occurs through Firefox telemetry, and standard telemetry opt-out methods apply.
No telemetry will be sent from the extension in the following additional cases:
* [Do Not Track](https://support.mozilla.org/en-US/kb/how-do-i-turn-do-not-track-feature) is enabled
* Preference: `privacy.donottrackheader.enabled`
* [Tracking Protection](https://support.mozilla.org/en-US/kb/tracking-protection) is enabled
* Preference: `privacy.trackingprotection.enabled`
* The user is in a [Private Browsing](https://support.mozilla.org/en-US/kb/private-browsing-use-firefox-without-history?redirectlocale=en-US&redirectslug=Private+Browsing) window
* Preference: `browser.privatebrowsing.autostart`
* [`windows.Window`](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/windows/Window) property: `window.incognito`
- [Do Not Track](https://support.mozilla.org/en-US/kb/how-do-i-turn-do-not-track-feature) is enabled
- Preference: `privacy.donottrackheader.enabled`
- [Tracking Protection](https://support.mozilla.org/en-US/kb/tracking-protection) is enabled
- Preference: `privacy.trackingprotection.enabled`
- The user is in a [Private Browsing](https://support.mozilla.org/en-US/kb/private-browsing-use-firefox-without-history?redirectlocale=en-US&redirectslug=Private+Browsing) window
- Preference: `browser.privatebrowsing.autostart`
- [`windows.Window`](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/windows/Window) property: `window.incognito`

Просмотреть файл

@ -18,10 +18,10 @@ import {handleWebRequest, updatePrices} from 'commerce/background/price_updates'
import store from 'commerce/state';
import {checkMigrations} from 'commerce/state/migrations';
import {loadStateFromStorage} from 'commerce/state/sync';
import telemetry from 'commerce/background/telemetry';
import {registerProbes} from 'commerce/background/telemetry';
(async function main() {
telemetry.registerProbes();
registerProbes();
// Set browser action default badge color, which can't be set via manifest
browser.browserAction.setBadgeBackgroundColor({

Просмотреть файл

@ -7,181 +7,176 @@
* @module
*/
const telemetry = {
// Scalar category names can't have underscores
CATEGORY: 'extension.pricealerts',
scalars: {
// Incremented when the user navigates to a supported site (e.g. Amazon)
supported_sites: {
kind: 'count',
record_on_release: true,
},
},
events: {
// User Events
// User clicks toolbar button to open the popup
open_popup: {
methods: ['open_popup'],
objects: ['toolbar_button'],
extra_keys: [
'badge_type',
'tracked_prods',
],
},
// User clicks on a UI element opening a page in a new tab
open_external_page: {
methods: ['open_external_page'],
objects: [
'amazon_link',
'best_buy_link',
'ebay_link',
'feedback_button',
'help_button',
'home_depot_link',
'learn_more_link',
'product_card',
'system_notification',
'walmart_link',
],
// For objects of type 'system_notification' and 'product' only ('tracked_prods' excepted)
extra_keys: [
'price',
'price_alert',
'price_orig',
'product_index',
'product_key',
'tracked_prods',
],
},
// User adds a product to the product listing
add_product: {
methods: ['add_product'],
objects: ['add_button'],
extra_keys: [
'price',
'product_key',
'tracked_prods',
],
},
// User deletes a product from the product listing
delete_product: {
methods: ['delete_product'],
objects: ['delete_button'],
extra_keys: [
'price',
'price_alert',
'price_orig',
'product_index',
'product_key',
'tracked_prods',
],
},
// User undeletes a product from the product listing
undo_delete_product: {
methods: ['undo_delete_product'],
objects: ['undo_button'],
extra_keys: [
'price',
'price_alert',
'price_orig',
'product_index',
'product_key',
'tracked_prods',
],
},
// User uninstalls the extension
uninstall: {
methods: ['uninstall'],
objects: ['uninstall'],
extra_keys: ['tracked_prods'],
},
// User hides the toolbar button for the extension
hide_toolbar_button: {
methods: ['hide_toolbar_button'],
objects: ['toolbar_button'],
extra_keys: [
'badge_type',
'tracked_prods',
],
},
// Scalar category names can't have underscores
const CATEGORY = 'extension.pricealerts';
// Non-User Events
// There is a price change on a tracked product
detect_price_change: {
methods: ['detect_price_change'],
objects: ['product_page'],
extra_keys: [
'price',
'price_orig',
'product_key',
'tracked_prods',
],
},
// Toolbar button is badged either due to a price alert or an extracted product
badge_toolbar_button: {
methods: ['badge_toolbar_button'],
objects: ['toolbar_button'],
extra_keys: [
'badge_type',
'tracked_prods',
],
},
// System notification is sent notifying user of a price alert
send_notification: {
methods: ['send_notification'],
objects: ['system_notification'],
extra_keys: [
'price',
'price_orig',
'product_key',
'tracked_prods',
],
},
// Product extraction is attempted on the content page
attempt_extraction: {
methods: ['attempt_extraction'],
objects: ['product_page'],
extra_keys: [
'extraction_id',
'is_bg_update',
'tracked_prods',
],
},
// Product extraction is completed on the content page
complete_extraction: {
methods: ['complete_extraction'],
objects: ['product_page'],
extra_keys: [
'extraction_id',
'is_bg_update',
'method',
'tracked_prods',
],
},
},
async registerProbes() {
await browser.telemetry.registerScalars(this.CATEGORY, this.scalars);
await browser.telemetry.registerEvents(this.CATEGORY, this.events);
},
async recordEvent(method, object, value, extra) {
if (!browser.telemetry.canUpload()) {
return;
}
await browser.telemetry.recordEvent(
this.CATEGORY,
method,
object,
value,
extra,
);
},
async scalarAdd(scalarName, value) {
await browser.telemetry.scalarAdd(
`${this.CATEGORY}.${scalarName}`,
value,
);
const SCALARS = {
// Incremented when the user navigates to a supported site (e.g. Amazon)
supported_sites: {
kind: 'count',
record_on_release: true,
},
};
export default telemetry;
const EVENTS = {
// User Events
// User clicks toolbar button to open the popup
open_popup: {
methods: ['open_popup'],
objects: ['toolbar_button'],
extra_keys: [
'badge_type',
'tracked_prods',
],
},
// User clicks on a UI element in the extension opening a page in a new tab
open_external_page: {
methods: ['open_external_page'],
objects: ['ui_element'],
extra_keys: [
'tracked_prods',
'element',
// For 'element' values 'product_card' and 'system_notification' only
'price',
'price_alert',
'price_orig',
'product_key',
// For 'element' value 'product_card' only
'product_index',
],
},
// User adds a product to the product listing
add_product: {
methods: ['add_product'],
objects: ['add_button'],
extra_keys: [
'price',
'product_key',
'tracked_prods',
],
},
// User deletes a product from the product listing
delete_product: {
methods: ['delete_product'],
objects: ['delete_button'],
extra_keys: [
'price',
'price_alert',
'price_orig',
'product_index',
'product_key',
'tracked_prods',
],
},
// User undeletes a product from the product listing
undo_delete_product: {
methods: ['undo_delete_product'],
objects: ['undo_button'],
extra_keys: [
'price',
'price_alert',
'price_orig',
'product_index',
'product_key',
'tracked_prods',
],
},
// User uninstalls the extension
uninstall: {
methods: ['uninstall'],
objects: ['uninstall'],
extra_keys: ['tracked_prods'],
},
// User hides the toolbar button for the extension
hide_toolbar_button: {
methods: ['hide_toolbar_button'],
objects: ['toolbar_button'],
extra_keys: [
'badge_type',
'tracked_prods',
],
},
// Non-User Events
// There is a price change on a tracked product
detect_price_change: {
methods: ['detect_price_change'],
objects: ['product_page'],
extra_keys: [
'price',
'price_orig',
'product_key',
'tracked_prods',
],
},
// Toolbar button is badged either due to a price alert or an extracted product
badge_toolbar_button: {
methods: ['badge_toolbar_button'],
objects: ['toolbar_button'],
extra_keys: [
'badge_type',
'tracked_prods',
],
},
// System notification is sent notifying user of a price alert
send_notification: {
methods: ['send_notification'],
objects: ['system_notification'],
extra_keys: [
'price',
'price_orig',
'product_key',
'tracked_prods',
],
},
// Product extraction is attempted on the content page
attempt_extraction: {
methods: ['attempt_extraction'],
objects: ['product_page'],
extra_keys: [
'extraction_id',
'is_bg_update',
'tracked_prods',
],
},
// Product extraction is completed on the content page
complete_extraction: {
methods: ['complete_extraction'],
objects: ['product_page'],
extra_keys: [
'extraction_id',
'is_bg_update',
'method',
'tracked_prods',
],
},
};
export async function registerProbes() {
await browser.telemetry.registerScalars(CATEGORY, SCALARS);
await browser.telemetry.registerEvents(CATEGORY, EVENTS);
}
export async function recordEvent(method, object, value, extra) {
if (!browser.telemetry.canUpload()) {
return;
}
await browser.telemetry.recordEvent(
CATEGORY,
method,
object,
value,
extra,
);
}
export async function scalarAdd(scalarName, value) {
if (!browser.telemetry.canUpload()) {
return;
}
await browser.telemetry.scalarAdd(
`${CATEGORY}.${scalarName}`,
value,
);
}

Просмотреть файл

@ -63,7 +63,7 @@ const CONFIG = {
browserActionUrl: new StringValue(browser.extension.getURL('/browser_action/index.html')),
// Price alert config
alertPercentThershold: new IntValue(5), // 5%
alertPercentThreshold: new IntValue(5), // 5%
alertAbsoluteThreshold: new IntValue(1000), // $10
/** Color of the toolbar badge for showing active price alerts. */