Bug 1763236 - add rudimentary documentation for mochitest-browser tests, r=sclements,hjones

Differential Revision: https://phabricator.services.mozilla.com/D142992

docs/config.yml

@@ -47,6 +47,7 @@ categories:
         - testing/intermittent
         - testing/testing-policy
         - testing/ci-configs
+        - testing/browser-chrome
         - testing/chrome-tests
         - testing/marionette
         - testing/geckodriver

testing/docs/browser-chrome/index.md

@@ -0,0 +1,89 @@
+Browser chrome mochitests
+=========================
+
+Browser chrome mochitests are mochitests that run in the context of the desktop
+Firefox browser window. The test files are named `browser_something.js` by
+convention, and in addition to mochitest assertions supports the
+[CommonJS standard assertions](http://wiki.commonjs.org/wiki/Unit_Testing/1.1),
+like [nodejs' assert module](https://nodejs.org/api/assert.html#assert) but
+implemented in [`Assert.jsm`](../assert.rst).
+
+These tests are used to test UI-related behaviour in Firefox for
+Desktop. They do not run on Android. If you're testing internal code that
+does not directly interact with the user interface,
+[xpcshell tests](../xpcshell/index.md) are probably a better fit for your needs.
+
+
+Running the tests
+-----------------
+
+You can run individual tests locally using the standard `./mach test` command:
+`./mach test path/to/browser_test.js`. You can omit the path if the filename
+is unique. You can also run entire directories, or specific test manifests:
+
+```
+./mach test path/to/browser.ini
+```
+
+You can also use the more specific `./mach mochitest` command in the same way.
+Using `./mach mochitest --help` will give you an exhaustive overview of useful
+other available flags relating to running, debugging and evaluating tests.
+
+For both commands, you can use the `--verify` flag to run the test under
+[test verification](../test-verification/index.rst). This helps flush out
+intermittent issues with the test.
+
+
+On our infrastructure, these tests run in the mochitest-browser-chrome jobs.
+There, they run on a per-manifest basis (so for most manifests, more than one
+test will run while the browser stays open).
+
+The tests also get run in `verify` mode in the `test-verify` jobs, whenever
+the test itself is changed.
+
+Note that these tests use "real" focus and input, so you'll need to not touch
+your machine while running them. You can run them with the `--headless`
+flag to avoid this, but some tests may break in this mode.
+
+
+Adding new tests
+----------------
+
+You can use the standard `./mach addtest path/to/new/browser_test.js` command
+to generate a new browser test, and add it to the relevant manifest, if tests
+already exist in that directory. This automatically creates a test file using
+the right template for you, and adds it to the manifest.
+
+If there are no tests in the directory yet (for example, for an entirely new
+feature and directory) you will need to:
+
+1. create an empty `browser.ini` file
+2. add it to `BROWSER_CHROME_MANIFESTS` collection from a `moz.build` file.
+3. then run the `./mach addtest` command as before.
+
+In terms of the contents of the test, please see [Writing new browser
+mochitests](writing.md).
+
+Debugging tests
+---------------
+
+The `./mach test` and `./mach mochitest` commands support a `--jsdebugger`
+flag which will open the browser toolbox. If you add the 
+[`debugger;` keyword](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/debugger)
+in your test, the debugger will pause there.
+
+Alternatively, you can set breakpoints using the debugger yourself. If you want
+to pause the debugger before running the test, you can use the `--no-autorun`
+flag. Alternatively, if you want to pause the debugger on failure, you can use
+`--debug-on-failure`.
+
+For more details, see [Avoiding intermittent tests](../intermittent/index.rst).
+
+Reference material
+------------------
+
+- [Assert module](../assert.rst)
+- [TestUtils module](../testutils.rst)
+- [BrowserTestUtils module](browsertestutils.rst)
+- [SimpleTest utilities](../simpletest.rst)
+- [EventUtils utilities](../eventutils.rst)

testing/docs/browser-chrome/writing.md

@@ -0,0 +1,148 @@
+# Writing new browser mochitests
+
+After [creating a new empty test file](index.html#adding-new-tests), you will
+have an empty `add_task` into which you can write your test.
+
+## General guidance
+
+The test can use `ok`, `is`, `isnot`, as well as all the regular
+[CommonJS standard assertions](http://wiki.commonjs.org/wiki/Unit_Testing/1.1),
+to make test assertions.
+
+The test can use `info` to log strings into the test output.
+``console.log`` will work for local runs of individual tests, but aren't
+normally used for checked-in tests.
+
+The test will run in a separate scope inside the browser window.
+`gBrowser`, `gURLBar`, `document`, and various other globals are thus
+accessible just as they are for non-test code in the same window. However,
+variables declared in the test file will not outlive the test.
+
+## Test architecture
+
+It is the responsibility of individual tests to leave the browser as they
+found it. If the test changes prefs, opens tabs, customizes the UI, or makes
+other changes, it should revert those when it is done.
+
+To help do this, a number of useful primitives are available:
+
+- `add_setup` allows you to add setup tasks that run before any `add_task` tasks.
+- `SpecialPowers.pushPrefEnv` ([see below](#changing-preferences)) allows you to set prefs that will be automatically
+  reverted when the test file has finished running.
+- [`BrowserTestUtils.withNewTab`](browsertestutils.html#BrowserTestUtils.withNewTab), allows you to easily run async code
+  talking to a tab that you open and close it when done.
+- `registerCleanupFunction` takes an async callback function that you can use
+  to do any other cleanup your test might need.
+
+## Common operations
+
+### Opening new tabs and new windows, and closing them
+
+Should be done using the relevant methods in `BrowserTestUtils` (which
+is available without any additional work).
+
+Typical would be something like:
+
+```lang=js
+add_task(async function() {
+  await BrowserTestUtils.withNewTab("https://example.com/mypage", async (browser) {
+    // `browser` will have finished loading the passed URL when this code runs.
+    // Do stuff with `browser` in here. When the async function exits,
+    // the test framework will clean up the tab.
+  });
+});
+```
+
+### Executing code in the content process associated with a tab or its subframes
+
+Should be done using `SpecialPowers.spawn`:
+
+```lang=js
+let result = await SpecialPowers.spawn(browser, [42, 100], async (val, val2) => {
+  // Replaces the document body with '42':
+  content.document.body.textContent = val;
+  // Optionally, return a result. Has to be serializable to make it back to
+  // the parent process (so DOM nodes or similar won't work!). 
+  return Promise.resolve(val2 * 2);
+});
+```
+
+You can pass a BrowsingContext reference instead of `browser` to directly execute
+code in subframes.
+
+Inside the function argument passed to `SpecialPowers.spawn`, `content` refers
+to the `window` of the web content in that browser/BrowsingContext.
+
+For some operations, like mouse clicks, convenience helpers are available on
+`BrowserTestUtils`:
+
+```lang=js
+await BrowserTestUtils.synthesizeMouseAtCenter("#my.css.selector", {accelKey: true}, browser);
+```
+
+### Changing preferences
+
+Use `SpecialPowers.pushPrefEnv`:
+
+```lang=js
+await SpecialPowers.pushPrefEnv({
+  set: [["accessibility.tabfocus", 7]]
+});
+```
+This example sets the pref allowing buttons and other controls to receive tab focus -
+this is the default on Windows and Linux but not on macOS, so it can be necessary in
+order for your test to pass reliably on macOS if it uses keyboard focus.
+
+### Wait for an observer service notification topic or DOM event
+
+Use the utilities for this on [`TestUtils`](../testutils.html#TestUtils.topicObserved):
+
+```lang=js
+await TestUtils.topicObserved("sync-pane-loaded");
+```
+
+and [`BrowserTestUtils`](browsertestutils.html#BrowserTestUtils.waitForEvent), respectively:
+
+```lang=js
+await BrowserTestUtils.waitForEvent(domElement, "click");
+```
+
+### Wait for some DOM to update.
+
+Use [`BrowserTestUtils.waitForMutationCondition`](browsertestutils.html#BrowserTestUtils.waitForMutationCondition).
+Do **not** use `waitForCondition`, which uses a timeout loop and often
+leads to intermittent failures.
+
+### Mocking code not under test
+
+The [`Sinon`](https://sinonjs.org/) mocking framework is available. You can import it
+using something like:
+
+```lang=js
+const { sinon } = ChromeUtils.import("resource://testing-common/Sinon.jsm");
+```
+
+More details on how to do mocking are available on the Sinon website.
+
+## Additional files
+
+You can use extra files (e.g. webpages to load) by adding them to a `support-files`
+property using the `browser.ini` file:
+
+```lang=ini
+[browser_foo.js]
+support-files =
+  bar.html
+  baz.js
+```
+
+## Reusing code across tests
+
+For operations that are common to a specific set of tests, you can use the `head.js`
+file to share JS code.
+
+Where code is needed across various directories of tests, you should consider if it's
+common enough to warrant being in `BrowserTestUtils.jsm`, or if not, setting up
+a separate `jsm` module containing your test helpers. You can add these to
+`TESTING_JS_MODULES` in `moz.build` to avoid packaging them with Firefox. They
+will be available in `resource://testing-common/` to all tests.