2019-01-29 18:18:42 +03:00
|
|
|
Testing
|
|
|
|
=======
|
|
|
|
|
2021-09-23 14:01:21 +03:00
|
|
|
The Remote Protocol has unit- and functional tests located under different folders:
|
2021-02-23 20:37:03 +03:00
|
|
|
|
2022-01-17 13:49:42 +03:00
|
|
|
- CDP: `remote/cdp`
|
|
|
|
- Marionette: `remote/marionette`.
|
|
|
|
- Shared Modules: `remote/shared/`
|
2021-09-23 14:01:21 +03:00
|
|
|
- WebDriver BiDi: `remote/webdriver-bidi`
|
2019-01-29 18:18:42 +03:00
|
|
|
|
2021-09-23 14:01:21 +03:00
|
|
|
You may want to run all the tests under a particular subfolder locally like this:
|
2019-01-29 18:18:42 +03:00
|
|
|
|
2019-04-25 13:49:31 +03:00
|
|
|
% ./mach test remote
|
2019-02-17 18:18:47 +03:00
|
|
|
|
|
|
|
|
|
|
|
Unit tests
|
|
|
|
----------
|
|
|
|
|
|
|
|
Because tests are run in parallel and [xpcshell] itself is quite
|
2019-01-29 18:18:42 +03:00
|
|
|
chatty, it can sometimes be useful to run the tests in sequence:
|
|
|
|
|
2021-02-23 20:37:03 +03:00
|
|
|
% ./mach xpcshell-test --sequential remote/cdp/test/unit/test_DomainCache.js
|
2019-01-29 18:18:42 +03:00
|
|
|
|
2019-04-25 13:49:31 +03:00
|
|
|
The unit tests will appear as part of the `X` (for _xpcshell_) jobs
|
|
|
|
on Treeherder.
|
2019-01-29 18:18:42 +03:00
|
|
|
|
2022-01-17 13:49:42 +03:00
|
|
|
[xpcshell]: /testing/xpcshell/index.rst
|
2019-02-17 18:18:47 +03:00
|
|
|
|
|
|
|
|
2022-01-17 13:49:42 +03:00
|
|
|
Browser Chrome Mochitests
|
|
|
|
-------------------------
|
2019-02-17 18:18:47 +03:00
|
|
|
|
2022-01-17 13:49:42 +03:00
|
|
|
We also have a set of functional browser-chrome mochitests located
|
|
|
|
under several components, ie. _remote/shared/messagehandler/test/browser_:
|
2019-02-17 18:18:47 +03:00
|
|
|
|
2022-01-17 13:49:42 +03:00
|
|
|
% ./mach mochitest remote/shared/messagehandler/test/browser/browser_*
|
2019-02-17 18:18:47 +03:00
|
|
|
|
2019-04-25 13:49:31 +03:00
|
|
|
The functional tests will appear under the `M` (for _mochitest_)
|
2019-10-19 00:01:44 +03:00
|
|
|
category in the `remote` jobs on Treeherder.
|
2019-04-25 13:49:31 +03:00
|
|
|
|
2019-04-29 16:24:33 +03:00
|
|
|
As the functional tests will sporadically pop up new Firefox
|
2022-01-17 13:49:42 +03:00
|
|
|
application windows, a helpful tip is to run them in headless
|
|
|
|
mode:
|
2019-04-29 16:24:33 +03:00
|
|
|
|
2022-01-17 13:49:42 +03:00
|
|
|
% ./mach mochitest --headless remote/shared/messagehandler/test/browser
|
2019-04-29 16:24:33 +03:00
|
|
|
|
|
|
|
The `--headless` flag is equivalent to setting the `MOZ_HEADLESS`
|
|
|
|
environment variable. You can additionally use `MOZ_HEADLESS_WIDTH`
|
|
|
|
and `MOZ_HEADLESS_HEIGHT` to control the dimensions of the virtual
|
|
|
|
display.
|
|
|
|
|
2022-01-17 13:49:42 +03:00
|
|
|
The `add_task()` function used for writing asynchronous tests is
|
2019-12-18 15:53:52 +03:00
|
|
|
replaced to provide some additional test setup and teardown useful
|
2021-05-25 12:13:29 +03:00
|
|
|
for writing tests against the Remote Agent and the targets.
|
2019-12-18 15:53:52 +03:00
|
|
|
|
2022-01-17 13:49:42 +03:00
|
|
|
There are also specific browser-chrome tests for CDP.
|
|
|
|
|
|
|
|
Before such a task is run, the `nsIRemoteAgent` listener is started
|
2019-12-18 15:53:52 +03:00
|
|
|
and a [CDP client] is connected. You will use this CDP client for
|
|
|
|
interacting with the agent just as any other CDP client would.
|
|
|
|
|
2020-01-03 00:13:20 +03:00
|
|
|
Also target discovery is getting enabled, which means that targetCreated,
|
|
|
|
targetDestroyed, and targetInfoChanged events will be received by the client.
|
|
|
|
|
2019-12-18 15:53:52 +03:00
|
|
|
The task function you provide in your test will be called with the
|
|
|
|
three arguments `client`, `CDP`, and `tab`:
|
|
|
|
|
|
|
|
- `client` is the connection to the `nsIRemoteAgent` listener,
|
|
|
|
and it provides the a client CDP API
|
|
|
|
|
|
|
|
- `CDP` is the CDP client class
|
|
|
|
|
|
|
|
- `tab` is a fresh tab opened for each new test, and is automatically
|
|
|
|
removed after the test has run
|
|
|
|
|
|
|
|
This is what it looks like all put together:
|
|
|
|
|
2020-01-13 11:52:44 +03:00
|
|
|
add_task(async function testName({client, CDP, tab}) {
|
2019-12-18 15:53:52 +03:00
|
|
|
// test tab is implicitly created for us
|
|
|
|
info("Current URL: " + tab.linkedBrowser.currentURI.spec);
|
|
|
|
|
|
|
|
// manually connect to a specific target
|
2021-06-01 22:41:01 +03:00
|
|
|
const { mainProcessTarget } = RemoteAgent.cdp.targetList;
|
2019-12-18 15:53:52 +03:00
|
|
|
const target = mainProcessTarget.wsDebuggerURL;
|
|
|
|
const client = await CDP({ target });
|
|
|
|
|
|
|
|
// retrieve the Browser domain, and call getVersion() on it
|
|
|
|
const { Browser } = client;
|
|
|
|
const version = await Browser.getVersion();
|
|
|
|
|
|
|
|
await client.close();
|
|
|
|
|
|
|
|
// tab is implicitly removed
|
|
|
|
});
|
|
|
|
|
|
|
|
You can control the tab creation behaviour with the `createTab`
|
|
|
|
option to `add_task(taskFunction, options)`:
|
|
|
|
|
2020-01-13 11:52:44 +03:00
|
|
|
add_task(async function testName({client}) {
|
2019-12-18 15:53:52 +03:00
|
|
|
// tab is not implicitly created
|
|
|
|
}, { createTab: false });
|
|
|
|
|
|
|
|
If you want to write an asynchronous test _without_ this implicit
|
|
|
|
setup you may instead use `add_plain_task()`, which works exactly like the
|
|
|
|
original `add_task()`.
|
|
|
|
|
|
|
|
[CDP client]: https://github.com/cyrus-and/chrome-remote-interface
|
2019-08-16 15:58:32 +03:00
|
|
|
|
|
|
|
|
|
|
|
Puppeteer tests
|
|
|
|
---------------
|
|
|
|
|
|
|
|
In addition to our own Firefox-specific tests, we run the upstream
|
2020-01-03 19:11:19 +03:00
|
|
|
[Puppeteer test suite] against our implementation to [track progress]
|
2020-06-05 21:53:38 +03:00
|
|
|
towards achieving full [Puppeteer support] in Firefox. The tests are written
|
|
|
|
in the behaviour-driven testing framework [Mocha].
|
2019-08-16 15:58:32 +03:00
|
|
|
|
2020-06-05 21:53:38 +03:00
|
|
|
Puppeteer tests are vendored under _remote/test/puppeteer/_ and are
|
2019-08-16 15:58:32 +03:00
|
|
|
run locally like this:
|
|
|
|
|
2020-06-05 21:53:38 +03:00
|
|
|
% ./mach puppeteer-test
|
|
|
|
|
|
|
|
You can also run them against Chrome as:
|
|
|
|
|
2022-03-15 10:10:36 +03:00
|
|
|
% ./mach puppeteer-test --product=chrome
|
|
|
|
|
|
|
|
By default the mach command will automatically install Puppeteer but that's
|
|
|
|
only needed for the very first time, or when a new Puppeteer release has been
|
|
|
|
vendored in. To skip the install step use the `--no-install` option.
|
|
|
|
|
|
|
|
To run only some specific tests from the whole test suite the appropriate
|
|
|
|
test files have to be updated first. To select specific tests or test
|
|
|
|
groups within a file define [exclusive tests] by adding the `.only` suffix
|
|
|
|
like `it.only()` or `describe.only()`. Make sure to specify as well the
|
|
|
|
`--subset` option to the mach command to disable the check for missing or
|
|
|
|
skipped tests in our log parsing.
|
2020-06-05 21:53:38 +03:00
|
|
|
|
2022-03-15 10:10:36 +03:00
|
|
|
More customizations for [Mocha] can be found in its own documentation.
|
2020-06-05 21:53:38 +03:00
|
|
|
|
2021-02-23 20:37:03 +03:00
|
|
|
Test expectation metadata is collected in _remote/test/puppeteer-expected.json_
|
2020-06-05 21:53:38 +03:00
|
|
|
via log parsing and a custom Mocha reporter under
|
|
|
|
_remote/test/puppeteer/json-mocha-reporter.js_
|
|
|
|
|
2022-03-15 10:10:36 +03:00
|
|
|
Check the upstream [Puppeteer test suite] documentation for instructions on
|
|
|
|
how to skip tests, run only one test or a subsuite of tests.
|
2021-09-23 14:01:21 +03:00
|
|
|
|
|
|
|
Testing on Try
|
|
|
|
--------------
|
|
|
|
|
|
|
|
To schedule all the Remote Protocol tests on try, you can use the
|
|
|
|
`remote-protocol` [try preset]:
|
|
|
|
|
|
|
|
mach try --preset remote-protocol
|
|
|
|
|
|
|
|
But you can also schedule tests by selecting relevant jobs yourself:
|
|
|
|
|
2021-10-05 11:06:54 +03:00
|
|
|
mach try fuzzy
|
2021-09-23 14:01:21 +03:00
|
|
|
|
2022-03-15 10:10:36 +03:00
|
|
|
[Puppeteer test suite]: https://github.com/puppeteer/puppeteer/blob/master/test/README.md
|
2019-08-16 15:58:32 +03:00
|
|
|
[Puppeteer support]: https://bugzilla.mozilla.org/show_bug.cgi?id=puppeteer
|
2020-01-03 19:11:19 +03:00
|
|
|
[Mocha]: https://mochajs.org/
|
2022-03-15 10:10:36 +03:00
|
|
|
[exclusive tests]: https://mochajs.org/#exclusive-tests
|
2022-01-17 13:49:42 +03:00
|
|
|
[track progress]: https://puppeteer.github.io/ispuppeteerfirefoxready/
|
|
|
|
[try preset]: /tools/try/presets
|