Improved docs in general

2019-03-01 23:45:44 +02:00 · 2019-03-01 23:45:44 +02:00 · b2ee54b4ec
--- a/README.md
+++ b/README.md
@ -84,3 +84,7 @@ Telemetry pings are loaded into the encrypted Pioneer pipeline.
 ## Improving this add-on

 See [DEV.md](./docs/DEV.md) for more details on how to work with this add-on as a developer.
+
+## References
+
+* [Bugzilla](https://bugzilla.mozilla.org/show_bug.cgi?id=TODO)
--- a/docs/DEV.md
+++ b/docs/DEV.md
@ -13,6 +13,7 @@
 * [Seeing the add-on in action](#seeing-the-add-on-in-action)
 * [Format code using prettier and eslint --fix](#format-code-using-prettier-and-eslint---fix)
 * [Generate stub code and API docs](#generate-stub-code-and-api-docs)
+* [Manual testing](#manual-testing)
 * [Automated testing](#automated-testing)
  * [Unit tests](#unit-tests)
  * [Functional tests](#functional-tests)
@ -115,7 +116,29 @@ npm run format
 npm run generate
 ```

-Generates stub code and API docs from `src/privileged/*/schema.yaml`. See <https://github.com/motin/webext-experiment-utils> for more information.
+Generates stub code and API docs from `src/privileged/*/schema.yaml` using <https://github.com/motin/webext-experiment-utils>.
+
+## Manual testing
+
+Launch the built add-on as already expired study:
+
+```shell
+EXPIRED=1 npm run test:manual
+```
+
+Launch the built add-on as expiring within a few seconds:
+
+```shell
+EXPIRE_IN_SECONDS=5 npm run test:manual
+```
+
+Launch the built add-on as with a specific variation set:
+
+```shell
+BRANCH=control npm run test:manual
+```
+
+For more options, see the code at [./run-firefox.js](./run-firefox.js).

 ## Automated testing

@ -153,23 +176,26 @@ ln -s Firefox\ Nightly.app FirefoxNightly.app
 npm run test:func
 ```

-Runs functional tests using the Selenium driver, verifying the telemetry payload at Firefox startup and add-on installation in a clean profile, then does **optimistic testing** of the _commonest path_ though the study for a user:
+Runs functional tests using the Selenium driver in a clean profile:

-* prove the notification bar ui opens
-* _clicking on the left-most button presented_.
-* verifying that sent Telemetry is correct.
+* `0-study_utils_integration.js` - Verifies the telemetry payload throughout the study life cycle.
+* `1-button_test.js` - _Example test_ Tests the browser action
+* `2-notification_bar.js` - _Example test_ Test the notification bar

 Code at [/test/functional/](/test/functional/).

 Note: The study variation/branch during tests is overridden by a preference in the FIREFOX_PREFERENCES section of `test/utils.js`.

-The functional testing set-up is a minimal set of tests imported from <https://github.com/mozilla/share-button-study> which contains plenty of examples of functional tests relevant to Shield study add-ons.
+The example tests is inspired by tests in <https://github.com/mozilla/share-button-study>.

 ## Directory Structure and Files

 ```
+├── .babelrc              # Used by karma to track code coverage in unit tests
 ├── .circleci             # Setup for .circle ci integration
-│   └── config.yml
+│   ├── config.yml
+│   └── reports
+│       └── .gitignore
 ├── .eslintignore
 ├── .eslintrc.js          # Linting configuration for mozilla, json etc
 ├── .gitignore
@ -177,45 +203,62 @@ The functional testing set-up is a minimal set of tests imported from <https://g
 ├── README.md
 ├── dist                  # Built zips (add-ons)
 │   ├── .gitignore
-│   └── button_icon_preference_study_shield_study_example_-2.0.0.zip
+│   └── button_icon_preference_-_shield_study_example-2.0.0.zip
 ├── docs
 │   ├── DEV.md
 │   ├── TELEMETRY.md      # Telemetry examples for this add-on
 │   ├── TESTPLAN.md       # Manual QA test plan
 │   └── WINDOWS_SETUP.md
+├── karma.conf.js
 ├── package-lock.json
 ├── package.json
+├── run-firefox.js
 ├── src                   # Files that will go into the add-on
-│   ├── .eslintrc.json
-│   ├── background.js     # Background scripts, independent of web pages or browser windows
-│   ├── icon.png
-│   ├── icons             # Icons used in the example study (remove in your add-on)
+│   ├── _locales
+│   │   ├── en-US
+│   │   │   └── messages.json
+│   │   └── fr
+│   │       └── messages.json
+│   ├── background.js     # Background script that contains a study life-cycle handler class and such boilerplate
+│   ├── feature.js        # Initiate your study logic using the Feature class in this file
+│   ├── icons
 │   │   ├── LICENSE
-│   │   ├── isolatedcorndog.svg
-│   │   ├── kittens.svg
-│   │   ├── lizard.svg
-│   │   └── puppers.svg
-│   ├── manifest.json     # The only file that every extension using WebExtension APIs must contain
-│   └── privileged
-│       ├── Config.jsm    # Study-specific configuration regarding branches, eligibility, expiration etc
-│       ├── feature
-│       │   ├── api.js
-│       │   ├── jsm
-│       │   │   └── Feature.jsm   # Contains study-specific privileged code
-│       │   └── schema.json
-│       └── shieldUtils
-│           ├── api.js
-│           ├── jsm
-│           │   ├── StudyUtils.jsm              # (copied in during `prebuild` and `prewatch`)
-│           │   └── StudyUtilsBootstrap.jsm     # Code from legacy bootstrap.js to be assimilated into StudyUtils
-│           └── schema.json
+│   │   ├── isolatedcorndog.svg # Icon used in the example study (remove in your add-on)
+│   │   ├── kittens.svg         # Icon used in the example study (remove in your add-on)
+│   │   ├── lizard.svg          # Icon used in the example study (remove in your add-on)
+│   │   ├── puppers.svg         # Icon used in the example study (remove in your add-on)
+│   │   ├── shield-icon.256.png
+│   │   ├── shield-icon.48.png
+│   │   ├── shield-icon.98.png
+│   │   └── shield-icon.svg
+│   ├── manifest.json     # The WebExtension manifest. Use this to declare permissions and web extension experiments etc
+│   ├── privileged
+│   │   ├── .gitignore
+│   │   ├── introductionNotificationBar
+│   │   │   ├── api.js
+│   │   │   └── schema.json
+│   │   ├── panel
+│   │   │   ├── blurts.EveryWindow.jsm
+│   │   │   └── cookiesrest.EveryWindow.js
+│   │   ├── privacyContext
+│   │   │   ├── api.js
+│   │   │   ├── api.md
+│   │   │   ├── schema.json
+│   │   │   ├── schema.yaml
+│   │   │   └── stubApi.js
+│   │   └── study
+│   │       ├── api.js
+│   │       └── schema.json
+│   └── studySetup.js
 └── test                  # Automated tests `npm test` and circle
-│   ├── Dockerfile
-│   ├── docker_setup.sh
-│   ├── functional_tests.js
-│   ├── test_harness.js
-│   ├── test_printer.py
-│   └── utils.js
+│   ├── .eslintrc.js
+│   ├── ensure_minimum_node_version.js
+│   ├── functional
+│   │   ├── 0-study_utils_integration.js
+│   │   ├── 1-button_test.js
+│   │   ├── 2-notification_bar.js
+│   │   └── utils.js
+│   ├── results           # Code coverage and log artifacts from test runs
 └── web-ext-config.js     # Configuration options used by the `web-ext` command

 >> tree -a -I 'node_modules|.git|.DS_Store'
--- a/docs/TESTPLAN.md
+++ b/docs/TESTPLAN.md
@ -10,7 +10,7 @@
  * [Preparations](#preparations)
  * [Install the add-on and enroll in the study](#install-the-add-on-and-enroll-in-the-study)
 * [Expected User Experience / Functionality](#expected-user-experience--functionality)
-  * [Do these tests](#do-these-tests)
+  * [Do these tests (in addition to ordinary regression tests)](#do-these-tests-in-addition-to-ordinary-regression-tests)
  * [Design](#design)
  * [Note: checking "sent Telemetry is correct"](#note-checking-sent-telemetry-is-correct)
 * [Debug](#debug)
@ -53,40 +53,53 @@ Icon will be the same every run.

 If the user clicks on the badge more than 3 times, it ends the study.

-### Do these tests
+### Surveys

-1. UI APPEARANCE. OBSERVE a notification bar with these traits:
+This study fires a survey at the following endings:

-   * Icon is 'heartbeat'
-   * Text is "Welcome to the new feature! Look for changes!",
-   * Clickable buttons with labels 'Thanks!' AND 'I do not want this.'
-   * An `x` button at the right that closes the notice.
+* `expired`

-   Test fails IF:
+### Do these tests (in addition to ordinary regression tests)

-   * There is no bar.
-   * Elements are not correct or are not displayed
+**UI appearance test 1**

-2. UI functionality: Thanks!
+* Install the add-on as per above
+* Verify that the study runs
+* OBSERVE a notification bar with these traits:
+  * Icon is 'heartbeat'
+  * Text is "Welcome to the new feature! Look for changes!",
+  * Clickable buttons with labels 'Thanks!' AND 'I do not want this.'
+  * An `x` button at the right that closes the notice.
+* Test fails IF:
+  * There is no bar.
+  * Elements are not correct or are not displayed

-   * Click on the 'Thanks!' button
-   * Verify that the notification bar closes
+**UI functionality test 1: Thanks!**

-3. UI functionality: I do not want this.
+* Install the add-on as per above
+* Verify that the study runs
+* Click on the 'Thanks!' button
+* Verify that the notification bar closes

-   * Click on the 'I do not want this.' button
-   * Verify that the notification bar closes
-   * Verify that the study ends
-   * Verify that sent Telemetry is correct
-   * Verify that the ending is `introduction-leave-study`
+**UI functionality test 2: I do not want this**

-4. UI functionality `too-popular`
+* Install the add-on as per above
+* Verify that the study runs
+* Click on the 'I do not want this.' button
+* Verify that the notification bar closes
+* Verify that the study ends
+* Verify that sent Telemetry is correct
+* Verify that the ending is `introduction-leave-study`

-   * Click on the web extension's icon three times
-   * Verify that the study ends
-   * Verify that sent Telemetry is correct
-   * Verify that the ending is `too-popular`
-   * Verify that the user is sent to the URL specified in `src/studySetup.js` under `endings -> too-popular`.
+**UI functionality test 3: `too-popular`**
+
+* Install the add-on as per above
+* Verify that the study runs
+* Click on the web extension's icon three times
+* Verify that the study ends
+* Verify that sent Telemetry is correct
+* Verify that the ending is `too-popular`
+* Verify that the user is sent to the URL specified in `src/studySetup.js` under `endings -> too-popular`.

 ### Design

--- a/run-firefox.js
+++ b/run-firefox.js
@ -17,7 +17,7 @@ const run = async studyType => {
    utils.FIREFOX_PREFERENCES,
  );
  const widgetId = utils.ui.makeWidgetId(
-    "jestr-pioneer-shield-study@pioneer.mozilla.org",
+    "extensions.button-icon-preference_shield_mozilla_org.test.variationName",
  );
  /*
  await utils.preferences.set(
--- a/src/feature.js
+++ b/src/feature.js
@ -1,6 +1,9 @@
 /* eslint no-unused-vars: ["error", { "varsIgnorePattern": "(feature)" }]*/

-/**  Example Feature module for a Shield Study.
+/**
+ * **Example template documentation - remove or replace this jsdoc in your study**
+ *
+ * Example Feature module for a Shield Study.
 *
 *  UI:
 *  - during INSTALL only, show a notification bar with 2 buttons:
@ -18,11 +21,10 @@
 **/
 class Feature {
  constructor() {}
-  /** A Demonstration feature.
-   *
-   *  - variation: study info about particular client study variation
-   *  - reason: string of background.js install/startup/shutdown reason
-   *
+
+  /**
+   * @param {Object} studyInfo Study info
+   * @returns {Promise<*>} Promise that resolves after configure
   */
  async configure(studyInfo) {
    const feature = this;
@ -77,11 +79,12 @@ class Feature {

  /**
   * Called at end of study, and if the user disables the study or it gets uninstalled by other means.
+   * @returns {Promise<*>} Promise that resolves after cleanup
   */
  async cleanup() {}

  /**
-   * Example of a utility function
+   * Example of a static utility function that can be unit-tested
   *
   * @param variation
   * @returns {string}
@ -91,6 +94,9 @@ class Feature {
  }
 }

+/**
+ * **Example browser action handling code - remove or replace in your study**
+ */
 class BrowserActionButtonChoiceFeature {
  /**
   * - set image, text, click handler (telemetry)