зеркало из https://github.com/mozilla/gecko-dev.git
207 строки
7.4 KiB
Markdown
207 строки
7.4 KiB
Markdown
Testing
|
||
=======
|
||
|
||
We verify and test Marionette in a couple of different ways, using
|
||
a combination of unit tests and functional tests. There are three
|
||
distinct components that we test:
|
||
|
||
- the Marionette **server**, using a combination of xpcshell
|
||
unit tests and functional tests written in Python spread across
|
||
Marionette- and WPT tests;
|
||
|
||
- the Python **client** is tested with the same body of functional
|
||
Marionette tests;
|
||
|
||
- and the **harness** that backs the Marionette, or `Mn` job on
|
||
try, tests is verified using separate mock-styled unit tests.
|
||
|
||
All these tests can be run by using [mach].
|
||
|
||
[mach]: https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/mach
|
||
|
||
xpcshell unit tests
|
||
-------------------
|
||
|
||
Marionette has a set of [xpcshell] unit tests located in
|
||
_testing/marionette/test/unit. These can be run this way:
|
||
|
||
% ./mach test testing/marionette/test/unit
|
||
|
||
Because tests are run in parallel and xpcshell itself is quite
|
||
chatty, it can sometimes be useful to run the tests sequentially:
|
||
|
||
% ./mach test --sequential testing/marionette/test/unit/test_error.js
|
||
|
||
These unit tests run as part of the `X` jobs on Treeherder.
|
||
|
||
[xpcshell]: https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests
|
||
|
||
|
||
Marionette functional tests
|
||
---------------------------
|
||
|
||
We also have a set of [functional tests] that make use of the Marionette
|
||
Python client. These start a Firefox process and tests the Marionette
|
||
protocol input and output, and will appear as `Mn` on Treeherder.
|
||
The following command will run all tests locally:
|
||
|
||
% ./mach marionette-test
|
||
|
||
But you can also run individual tests:
|
||
|
||
% ./mach marionette-test testing/marionette/harness/marionette_harness/tests/unit/test_navigation.py
|
||
|
||
In case you want to run the tests with another binary like [Firefox Nightly]:
|
||
|
||
% ./mach marionette-test --binary /path/to/nightly/firefox TEST
|
||
|
||
When working on Marionette it is often useful to surface the stdout
|
||
from Gecko, which can be achived using the `--gecko-log` option.
|
||
See <Debugging.html> for usage instructions, but the gist is that
|
||
you can redirect all Gecko output to stdout:
|
||
|
||
% ./mach marionette-test --gecko-log - TEST
|
||
|
||
Our functional integration tests pop up Firefox windows sporadically,
|
||
and a helpful tip is to suppress the window can be to use Firefox’
|
||
[headless mode]:
|
||
|
||
% ./mach marionette-test -z TEST
|
||
|
||
`-z` is an alias for the `--headless` flag and equivalent to setting
|
||
the `MOZ_HEADLESS` output variable. In addition to `MOZ_HEADLESS`
|
||
there is also `MOZ_HEADLESS_WIDTH` and `MOZ_HEADLESS_HEIGHT` for
|
||
controlling the dimensions of the no-op virtual display. This is
|
||
similar to using Xvfb(1) which you may know from the X windowing system,
|
||
but has the additional benefit of also working on macOS and Windows.
|
||
|
||
[functional tests]: PythonTests.html
|
||
[Firefox Nightly]: https://nightly.mozilla.org/
|
||
|
||
|
||
### Android
|
||
|
||
Prerequisites:
|
||
|
||
* You have [built Fennec](https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Firefox_for_Android_build).
|
||
* You can run an Android [emulator](https://wiki.mozilla.org/Mobile/Fennec/Android/Testing#Running_tests_on_the_Android_emulator),
|
||
which means you have the AVD you need.
|
||
|
||
When running tests on Fennec, you can have Marionette runner take care of
|
||
starting Fennec and an emulator, as shown below.
|
||
|
||
% ./mach marionette-test --emulator --app fennec
|
||
--avd-home /path/to/.mozbuild/android-device/avd
|
||
--emulator-binary /path/to/.mozbuild/android-sdk/emulator/emulator
|
||
--avd=mozemulator-x86
|
||
|
||
For Fennec tests, if the appropriate `emulator` command is in your `PATH`, you may omit the `--emulator-binary` argument. See `./mach marionette-test -h`
|
||
for additional options.
|
||
|
||
Alternately, you can start an emulator yourself and have the Marionette runner
|
||
start Fennec for you:
|
||
|
||
% ./mach marionette-test --emulator --app='fennec' --address=127.0.0.1:2828
|
||
|
||
To connect to an already-running Fennec in an already running emulator or on a device, you will need to enable Marionette manually by setting the browser preference
|
||
`marionette.enabled` set to true in the Fennec profile.
|
||
|
||
Make sure port 2828 is forwarded:
|
||
|
||
% adb forward tcp:2828 tcp:2828
|
||
|
||
If Fennec is already started:
|
||
|
||
% ./mach marionette-test --app='fennec' --address=127.0.0.1:2828
|
||
|
||
If Fennec is not already started on the emulator/device, add the `--emulator`
|
||
option. Marionette Test Runner will take care of forwarding the port and
|
||
starting Fennec with the correct prefs. (You may need to run
|
||
`adb forward --remove-all` to allow the runner to start.)
|
||
|
||
% ./mach marionette-test --emulator --app='fennec' --address=127.0.0.1:2828 --disable-e10s
|
||
--startup-timeout=300
|
||
|
||
If you need to troubleshoot the Marionette connection, the most basic check is
|
||
to start Fennec, make sure the `marionette.enabled` browser preference is
|
||
true and port 2828 is forwarded, then see if you get any response from
|
||
Marionette when you connect manually:
|
||
|
||
% telnet 127.0.0.1:2828
|
||
|
||
You should see output like `{"applicationType":"gecko","marionetteProtocol":3}`
|
||
|
||
[headless mode]: https://developer.mozilla.org/en-US/Firefox/Headless_mode
|
||
[geckodriver]: /testing/geckodriver/geckodriver
|
||
|
||
|
||
WPT functional tests
|
||
--------------------
|
||
|
||
Marionette is also indirectly tested through [geckodriver] with WPT
|
||
(`Wd` on Treeherder). To run them:
|
||
|
||
% ./mach wpt testing/web-platform/tests/webdriver
|
||
|
||
WPT tests conformance to the [WebDriver] standard and uses
|
||
[geckodriver]. Together with the Marionette remote protocol in
|
||
Gecko, they make up Mozilla’s WebDriver implementation.
|
||
|
||
This command supports a `--webdriver-arg '-vv'` argument that
|
||
enables more detailed logging, as well as `--jsdebugger` for opening
|
||
the Browser Toolbox.
|
||
|
||
A particularly useful trick is to combine this with the [headless
|
||
mode] for Firefox we learned about earlier:
|
||
|
||
% MOZ_HEADLESS=1 ./mach wpt --webdriver-arg '-vv' testing/web-platform/tests/webdriver
|
||
|
||
[WebDriver]: https://w3c.github.io/webdriver/webdriver-spec.html
|
||
|
||
|
||
Harness tests
|
||
-------------
|
||
|
||
The Marionette harness Python package has a set of mock-styled unit
|
||
tests that uses the [pytest] framework. The following command will
|
||
run all tests:
|
||
|
||
% ./mach python-test testing/marionette
|
||
|
||
To run a specific test specify the full path to the module:
|
||
|
||
% ./mach python-test testing/marionette/harness/marionette_harness/tests/harness_unit/test_serve.py
|
||
|
||
[pytest]: https://docs.pytest.org/en/latest/
|
||
|
||
|
||
One-click loaners
|
||
-----------------
|
||
|
||
Additionally, for debugging hard-to-reproduce test failures in CI,
|
||
one-click loaners from <Taskcluster.html> can be particularly useful.
|
||
|
||
|
||
Out-of-tree testing
|
||
-------------------
|
||
|
||
All the above examples show tests running _in-tree_, with a local
|
||
checkout of _central_ and a local build of Firefox. It is also
|
||
possibly to run the Marionette tests _without_ a local build and
|
||
with a downloaded test archive from <Taskcluster.html>.
|
||
|
||
If you want to run tests from a downloaded test archive, you will
|
||
need to download the `target.common.tests.tar.gz` artifact attached to
|
||
Treeherder [build jobs] `B` for your system. Extract the archive
|
||
and set up the Python Marionette client and harness by executing
|
||
the following command in a virtual environment:
|
||
|
||
% pip install -r config/marionette_requirements.txt
|
||
|
||
The tests can then be found under
|
||
_marionette/tests/testing/marionette/harness/marionette_harness/tests_
|
||
and can be executed with the command `marionette`. It supports
|
||
the same options as described above for `mach`.
|
||
|
||
[build jobs]: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=build
|