This commit prepares the decks for turning specific Raptor tasks into
Raptor + browsertime tasks. The `--browsertime` flag to `mach try
...` flips the switch; eventually, the Raptor harness will recognize
the `--browsertime` flag and use browsertime to perform the pageload
measurements.
To run browsertime, we need:
1) Node.js
2) the browsertime `node_modules` (provided by the
`toolchain-browsertime` task)
3) ffmpeg (for producing videos from captured frames)
4) chromedriver (in the future, when targeting Chrome/Chromium)
5) geckodriver (provided by the `toolchain-*-geckodriver` tasks)
6) `PATH` configured
This commit arranges those things.
Since the configuration varies by test platform, and eventually we
expect the changes implemented by the flag to be moved into YAML task
definitions, we elect to use `by-test-platform` conditionals as much
as possible. The end expression is pleasant, thanks to
`evaluate_keyed_by`.
Handling PATH, however, is a rabbit hole. At this time, it's not
possible to use `fetch` task repackaging, because `releng-hardware`
doesn't support `zstandard` (Bug 1576244) and there's no appetite to
avoid `zstandard` entirely (Bug 1576698). Generally PATH is
configured using `mozharness` configuration files, which can execute
arbitrary Python and configure the PATH only for browsertime jobs.
However, the Raptor mozharness script itself runs the Raptor harness
in a stripped down environment, throwing away modifications to PATH.
It's not clear what impacts changing that has, so we leave it alone,
and add a `--browsertime-ffmpeg` flag and custom handling in the
Raptor harness. This can transition smoothly into a browsertime flag
(so that the PATH doesn't need to be set at all) and into a unified
interface for Raptor and `mach browsertime` to configure the
browsertime execution environment.
Differential Revision: https://phabricator.services.mozilla.com/D38781
--HG--
extra : moz-landing-system : lando
This patch adds support for using `./mach raptor` instead of `./mach raptor-test`. The old version is left for backwards compatibility.
Differential Revision: https://phabricator.services.mozilla.com/D43051
--HG--
extra : moz-landing-system : lando
Changed the required yml, ini, json, js and html files to add Jetstream2 benchmark test to Raptor.
The changes on the javascript files are on github.
Differential Revision: https://phabricator.services.mozilla.com/D35418
--HG--
extra : moz-landing-system : lando
The goal is to configure browsertime in Raptor in two ways:
1) locally, just like `mach browsertime` does;
2) in automation, at taskgraph creation time, using fetches and
mozharness suite artifacts (for geckodriver).
It's possible for this to be done using mozharness config settings but
using command line options is more explicit and more likely to be easy
to remove later if and when we transition to a browsertime-specific
mozharness script.
Differential Revision: https://phabricator.services.mozilla.com/D38776
--HG--
extra : moz-landing-system : lando
mozboot.util.get_state_dir() returns a tuple of (<path>, <bool). The bool
denotes whether or not the state dir came from an environment variable.
But this value is only used in a single place, and is very easy to test for
anyway. It's not worth the added complexity it imposes on all other consumers
of this function. Let's just make this function return the path.
Differential Revision: https://phabricator.services.mozilla.com/D15723
--HG--
extra : moz-landing-system : lando
Rap-P treeherder group for Raptor power tests.
Emit separate PERFHERDER_DATA for power in addition to the performance measurements.
Use magic --host HOST_IP value to have framework load host ip from environment
variable HOST_IP.
This fixes a bug where the benchmarks weren't being copied properly on Windows.
While it is slightly slower to add every file to the manifest (~0.2 seconds), it
still outperforms shutil.copytree and is more robust (will use links on Linux/Mac
while using copy on Windows).
Differential Revision: https://phabricator.services.mozilla.com/D3234
--HG--
extra : moz-landing-system : lando
Currently the benchmarks raptor runs are checked into the tree at:
third_party/webkit/PerformanceTests
However, some benchmarks are too big to vendor (200+MB). This patch will
clone a repo containing the benchmarks that *aren't* checked into the tree,
into the user's ~/.mozbuild directory. If the repo already exists, an update
will be run.
Differential Revision: https://phabricator.services.mozilla.com/D2306
--HG--
extra : moz-landing-system : lando
Includes a temporary downloading/installing Chrome 'hack' just so we can get going on try, until 'official' chrome builds are available in CI. Also some small raptor fixes/changes to support speedometer on chrome.
MozReview-Commit-ID: 8ZBlOg0BOE4
--HG--
extra : rebase_source : 8271a7b9efd6b1d30e363eb47876d82f243d5ca7
Botond Ballo
Rob Wood
Bogdan Tara
Nick Alexander
Gregory Mierzwinski
Marian Raiciof
Chris Hartjes
Henrik Skupin
Ivan Yung
Stephen Donner
Bob Clary
Ionut Goldan
Florin Strugariu
Cosmin Sabou
Andrew Halberstadt
Geoff Brown