This patch adds the ability to check if all subtests were run and ensures that we fail the task when at least one test is missing from the expected set of tests. It also modifies how the return code is computed in output.output to check that all checks passed and that none of them failed.
Differential Revision: https://phabricator.services.mozilla.com/D47630
--HG--
extra : moz-landing-system : lando
This patch adds fullscreen and windowed youtube tests for the V9 and H264 encoding at 1080p30 and 1080p60. Each subtest runs for 20 page cycles which amounts to about 5 minutes each. It also begins adding these to power test tasks running on the macosx-1014 reference hardware.
Differential Revision: https://phabricator.services.mozilla.com/D45067
--HG--
extra : moz-landing-system : lando
This patch re-adds code that was removed by the patch in bug 1565316 - the summarize_supporting_data function. It also fixes the construct_summary function to accept a unit argument for summarizing the values in supporting data measurements.
Differential Revision: https://phabricator.services.mozilla.com/D46995
--HG--
extra : moz-landing-system : lando
This patch adds google chrome release tests for windows10-64, windows7-32, linux64, and macosx. It will run anywhere chromium is currently running, and uses the same settings as chromium for tier, max-run-time, etc.. All chromium test tasks will remain as they are - they will be run in a cron task in the future.
Differential Revision: https://phabricator.services.mozilla.com/D45385
--HG--
extra : moz-landing-system : lando
Stop running all Raptor tests that run against Fennec. Raptor tests running
against geckoview and geckoview products continues.
Differential Revision: https://phabricator.services.mozilla.com/D44511
--HG--
extra : moz-landing-system : lando
This task has been added to create all the missing cold tests for mobile websites.
Differential Revision: https://phabricator.services.mozilla.com/D45634
--HG--
extra : moz-landing-system : lando
This patch modifies how CPU usage is measured. First, it swaps out `dumpsys cpuinfo` in favor of `top` for android 7 devices. Next, a new class AndroidCpuProfiler is created for CPU polling since `top` only returns a snapshot and is not representative of what the browser has been doing (previously, we only measured CPU usage before the test started).
The CPU profiler/poller is started where the snapshot used to be generated and the perfherder data is now generated before the browser shuts down to prevent recording noise from the shutdown. The poller gathers a data point every 10 seconds, but this can be modified. It then produces perfherder-data for the average, minimum, and maximum CPU usage over the course of the test. The existing CPU tests have also been modified to accomodate these changes.
Differential Revision: https://phabricator.services.mozilla.com/D44919
--HG--
extra : moz-landing-system : lando
This commit prepares the decks for turning specific Raptor tasks into
Raptor + browsertime tasks. The `--browsertime` flag to `mach try
...` flips the switch; eventually, the Raptor harness will recognize
the `--browsertime` flag and use browsertime to perform the pageload
measurements.
To run browsertime, we need:
1) Node.js
2) the browsertime `node_modules` (provided by the
`toolchain-browsertime` task)
3) ffmpeg (for producing videos from captured frames)
4) chromedriver (in the future, when targeting Chrome/Chromium)
5) geckodriver (provided by the `toolchain-*-geckodriver` tasks)
6) `PATH` configured
This commit arranges those things.
Since the configuration varies by test platform, and eventually we
expect the changes implemented by the flag to be moved into YAML task
definitions, we elect to use `by-test-platform` conditionals as much
as possible. The end expression is pleasant, thanks to
`evaluate_keyed_by`.
Handling PATH, however, is a rabbit hole. At this time, it's not
possible to use `fetch` task repackaging, because `releng-hardware`
doesn't support `zstandard` (Bug 1576244) and there's no appetite to
avoid `zstandard` entirely (Bug 1576698). Generally PATH is
configured using `mozharness` configuration files, which can execute
arbitrary Python and configure the PATH only for browsertime jobs.
However, the Raptor mozharness script itself runs the Raptor harness
in a stripped down environment, throwing away modifications to PATH.
It's not clear what impacts changing that has, so we leave it alone,
and add a `--browsertime-ffmpeg` flag and custom handling in the
Raptor harness. This can transition smoothly into a browsertime flag
(so that the PATH doesn't need to be set at all) and into a unified
interface for Raptor and `mach browsertime` to configure the
browsertime execution environment.
Differential Revision: https://phabricator.services.mozilla.com/D38781
--HG--
extra : moz-landing-system : lando
This patch integrates mozpower into raptor desktop testing. It can be used for MacOS power testing by supplying the `--power-test` command at the command line.
Some changes to how the `--power-test` command is parsed are also made so that we don't check for the `--host` argument when it is supplied. Now, it is only checked when `--app` is an android browser, in the near future this `--host` argument will no longer be needed for power testing on android.
Another change in this patch is the addition of the `self.artifact_dir` property which returns the current directory that should be used to output the artifacts.
Power usage data is output into this artifact directory in a 'power-measurements' folder and this data is zipped when we are running in CI.
Differential Revision: https://phabricator.services.mozilla.com/D42014
--HG--
extra : moz-landing-system : lando
This patch adds support for using `./mach raptor` instead of `./mach raptor-test`. The old version is left for backwards compatibility.
Differential Revision: https://phabricator.services.mozilla.com/D43051
--HG--
extra : moz-landing-system : lando
This patch fixes an issue where if browser_cycle does not equal 1 for the first entry in the raptor results, then it will attempt to access combined_suites with a key entry which would only get created when we had an entry with browser_cycle equal to 1. With this change, we use the suite name to determine when we need to initialize the combined_suites entry. This issue occurs when the first cycle does not produce results.
Differential Revision: https://phabricator.services.mozilla.com/D43060
--HG--
extra : moz-landing-system : lando
The alert_on values from the test INI were not being passed into the test settings json via manifest.py, this patch will fix this.
Differential Revision: https://phabricator.services.mozilla.com/D43049
--HG--
extra : moz-landing-system : lando
This requires generalizing the existing flags from paths to possibly
being paths. The `os.path.expandvars` allows to refer to
`MOZ_FETCHES_DIR` symbolically in the build scripts (which aren't
invoked by the shell before execution, leaving us to expand shell
variables).
Differential Revision: https://phabricator.services.mozilla.com/D41604
--HG--
extra : moz-landing-system : lando
This picks up various improvements, especially to how configurations are handled and some new rules.
Differential Revision: https://phabricator.services.mozilla.com/D37271
--HG--
extra : moz-landing-system : lando
This patch adds two new test tasks for each of the 4 Android browsers running speedometer, idle, or idle-background. These tests gather (i) cpu+memory usage, or (ii) cpu+memory+power usage information. The power tests are left as they are for now to determine if there is an impact to our measurements if we run all resource-usage tests at once. Furthermore, the cron job for android power tests was also modified to prevent it from picking up these tasks.
The cpu usage functons needed to be fixed here because they were not working on MotoG5/Android7 for two reasons: (i) device version was not being detected properly, and (ii) ADB shell commands can't use the pipe functionality (`|`). Tests needed to be modified to compensate for these changes as well.
Differential Revision: https://phabricator.services.mozilla.com/D42679
--HG--
extra : moz-landing-system : lando
Based on the conclusion from the bug mentioned above, we will run youtube playback always muted on desktop.
Differential Revision: https://phabricator.services.mozilla.com/D41555
--HG--
extra : moz-landing-system : lando
This patch disables adaptive brightness on any android device that runs power tests. It is not restored because it should always be disabled.
Differential Revision: https://phabricator.services.mozilla.com/D41444
--HG--
extra : moz-landing-system : lando
Florin Strugariu
Rob Wood
Bogdan Tara
Andrew Halberstadt
Gregory Mierzwinski
Nick Alexander
Marian Raiciof
octavian.negru
Tarek Ziadé
Geoff Brown
stephendonner
Mihai Alexandru Michis
Mark Banner
krubin
chrmod
Andrew Creskey