This task has been added to create all the missing cold tests for mobile websites.
Differential Revision: https://phabricator.services.mozilla.com/D45634
--HG--
extra : moz-landing-system : lando
This patch modifies how CPU usage is measured. First, it swaps out `dumpsys cpuinfo` in favor of `top` for android 7 devices. Next, a new class AndroidCpuProfiler is created for CPU polling since `top` only returns a snapshot and is not representative of what the browser has been doing (previously, we only measured CPU usage before the test started).
The CPU profiler/poller is started where the snapshot used to be generated and the perfherder data is now generated before the browser shuts down to prevent recording noise from the shutdown. The poller gathers a data point every 10 seconds, but this can be modified. It then produces perfherder-data for the average, minimum, and maximum CPU usage over the course of the test. The existing CPU tests have also been modified to accomodate these changes.
Differential Revision: https://phabricator.services.mozilla.com/D44919
--HG--
extra : moz-landing-system : lando
This commit prepares the decks for turning specific Raptor tasks into
Raptor + browsertime tasks. The `--browsertime` flag to `mach try
...` flips the switch; eventually, the Raptor harness will recognize
the `--browsertime` flag and use browsertime to perform the pageload
measurements.
To run browsertime, we need:
1) Node.js
2) the browsertime `node_modules` (provided by the
`toolchain-browsertime` task)
3) ffmpeg (for producing videos from captured frames)
4) chromedriver (in the future, when targeting Chrome/Chromium)
5) geckodriver (provided by the `toolchain-*-geckodriver` tasks)
6) `PATH` configured
This commit arranges those things.
Since the configuration varies by test platform, and eventually we
expect the changes implemented by the flag to be moved into YAML task
definitions, we elect to use `by-test-platform` conditionals as much
as possible. The end expression is pleasant, thanks to
`evaluate_keyed_by`.
Handling PATH, however, is a rabbit hole. At this time, it's not
possible to use `fetch` task repackaging, because `releng-hardware`
doesn't support `zstandard` (Bug 1576244) and there's no appetite to
avoid `zstandard` entirely (Bug 1576698). Generally PATH is
configured using `mozharness` configuration files, which can execute
arbitrary Python and configure the PATH only for browsertime jobs.
However, the Raptor mozharness script itself runs the Raptor harness
in a stripped down environment, throwing away modifications to PATH.
It's not clear what impacts changing that has, so we leave it alone,
and add a `--browsertime-ffmpeg` flag and custom handling in the
Raptor harness. This can transition smoothly into a browsertime flag
(so that the PATH doesn't need to be set at all) and into a unified
interface for Raptor and `mach browsertime` to configure the
browsertime execution environment.
Differential Revision: https://phabricator.services.mozilla.com/D38781
--HG--
extra : moz-landing-system : lando
This patch integrates mozpower into raptor desktop testing. It can be used for MacOS power testing by supplying the `--power-test` command at the command line.
Some changes to how the `--power-test` command is parsed are also made so that we don't check for the `--host` argument when it is supplied. Now, it is only checked when `--app` is an android browser, in the near future this `--host` argument will no longer be needed for power testing on android.
Another change in this patch is the addition of the `self.artifact_dir` property which returns the current directory that should be used to output the artifacts.
Power usage data is output into this artifact directory in a 'power-measurements' folder and this data is zipped when we are running in CI.
Differential Revision: https://phabricator.services.mozilla.com/D42014
--HG--
extra : moz-landing-system : lando
This patch adds support for using `./mach raptor` instead of `./mach raptor-test`. The old version is left for backwards compatibility.
Differential Revision: https://phabricator.services.mozilla.com/D43051
--HG--
extra : moz-landing-system : lando
This patch fixes an issue where if browser_cycle does not equal 1 for the first entry in the raptor results, then it will attempt to access combined_suites with a key entry which would only get created when we had an entry with browser_cycle equal to 1. With this change, we use the suite name to determine when we need to initialize the combined_suites entry. This issue occurs when the first cycle does not produce results.
Differential Revision: https://phabricator.services.mozilla.com/D43060
--HG--
extra : moz-landing-system : lando
The alert_on values from the test INI were not being passed into the test settings json via manifest.py, this patch will fix this.
Differential Revision: https://phabricator.services.mozilla.com/D43049
--HG--
extra : moz-landing-system : lando
This requires generalizing the existing flags from paths to possibly
being paths. The `os.path.expandvars` allows to refer to
`MOZ_FETCHES_DIR` symbolically in the build scripts (which aren't
invoked by the shell before execution, leaving us to expand shell
variables).
Differential Revision: https://phabricator.services.mozilla.com/D41604
--HG--
extra : moz-landing-system : lando
This picks up various improvements, especially to how configurations are handled and some new rules.
Differential Revision: https://phabricator.services.mozilla.com/D37271
--HG--
extra : moz-landing-system : lando
This patch adds two new test tasks for each of the 4 Android browsers running speedometer, idle, or idle-background. These tests gather (i) cpu+memory usage, or (ii) cpu+memory+power usage information. The power tests are left as they are for now to determine if there is an impact to our measurements if we run all resource-usage tests at once. Furthermore, the cron job for android power tests was also modified to prevent it from picking up these tasks.
The cpu usage functons needed to be fixed here because they were not working on MotoG5/Android7 for two reasons: (i) device version was not being detected properly, and (ii) ADB shell commands can't use the pipe functionality (`|`). Tests needed to be modified to compensate for these changes as well.
Differential Revision: https://phabricator.services.mozilla.com/D42679
--HG--
extra : moz-landing-system : lando
Based on the conclusion from the bug mentioned above, we will run youtube playback always muted on desktop.
Differential Revision: https://phabricator.services.mozilla.com/D41555
--HG--
extra : moz-landing-system : lando
This patch disables adaptive brightness on any android device that runs power tests. It is not restored because it should always be disabled.
Differential Revision: https://phabricator.services.mozilla.com/D41444
--HG--
extra : moz-landing-system : lando
Changed the required yml, ini, json, js and html files to add Jetstream2 benchmark test to Raptor.
The changes on the javascript files are on github.
Differential Revision: https://phabricator.services.mozilla.com/D35418
--HG--
extra : moz-landing-system : lando
This patch fixes four things at once:
(1) Consolidate supporting raptor measurements into one PERFHERDER_DATA output per type. For example, with this change, all power data will be grouped into one PERFHERDER_DATA output.
(2) Output perfherder-data-<DATA_TYPE>.json for each supporting measurement instead of overwriting perfherder-data.json which contains the regular raptor test results.
(3) Take an average of the supporting measurements when particular unit's are specified. In this case, the '%' unit makes us take the average instead of the sum of the measurements.
(4) Remove the redundant test name entry that prefixes all power subtest entries.
Differential Revision: https://phabricator.services.mozilla.com/D40667
--HG--
extra : moz-landing-system : lando
Add functionality for being able to send extra parameters for the test_url query of a test, directly from a taskcluster config.
Also, the PR adds logic in the `setup-raptor` taskgraph transform for dynamically changing the subset of youtube-playback tests based on the platform and project
Differential Revision: https://phabricator.services.mozilla.com/D39006
--HG--
extra : moz-landing-system : lando
The goal is to configure browsertime in Raptor in two ways:
1) locally, just like `mach browsertime` does;
2) in automation, at taskgraph creation time, using fetches and
mozharness suite artifacts (for geckodriver).
It's possible for this to be done using mozharness config settings but
using command line options is more explicit and more likely to be easy
to remove later if and when we transition to a browsertime-specific
mozharness script.
Differential Revision: https://phabricator.services.mozilla.com/D38776
--HG--
extra : moz-landing-system : lando
This mechanical transform lays the foundation for generalizing the
underlying test runner.
Differential Revision: https://phabricator.services.mozilla.com/D37132
--HG--
extra : moz-landing-system : lando
This mechanical transform lays the foundation for generalizing the
underlying test runner.
Differential Revision: https://phabricator.services.mozilla.com/D37132
--HG--
extra : moz-landing-system : lando
This mechanical transform lays the foundation for generalizing the
underlying test runner.
Differential Revision: https://phabricator.services.mozilla.com/D37132
--HG--
extra : moz-landing-system : lando
Bug 1547135 added functionality to configure specific Android devices
for performance tests. That code landed as part of device setup in
`testing/raptor/raptor.py` but there is nothing Raptor-specific
(indeed, nothing harness-specific) about this configuration.
Rather than lift it to `mozdevice`, this commit moves it into a
`performance_tuning` module. That's enough to slim down `raptor.py`.
Differential Revision: https://phabricator.services.mozilla.com/D37128
--HG--
extra : moz-landing-system : lando
Handle the exception ADBProcessError raised by
thermal_zone0 = self.device.shell_output('cat sys/class/thermal/thermal_zone0/temp')
on the emulator
Differential Revision: https://phabricator.services.mozilla.com/D38859
--HG--
extra : moz-landing-system : lando
Added the crash method for RaptorLogger class,
which will call structuredlog.py crash method.
Differential Revision: https://phabricator.services.mozilla.com/D38055
--HG--
extra : moz-landing-system : lando
This patch adds an additional data output to android power tests. This data is the power usage of the test calculated as a percentage increase relative to the OS baseline. test_power.py needed to be changed to accommodate these changes as well.
Differential Revision: https://phabricator.services.mozilla.com/D37462
--HG--
extra : moz-landing-system : lando
Marian Raiciof
Florin Strugariu
Tarek Ziadé
stephendonner
Mihai Alexandru Michis
Nick Alexander
Gregory Mierzwinski
Rob Wood
Mark Banner
krubin
chrmod
Andrew Creskey
Alexandru Irimovici
Narcis Beleuzu
Gijs Kruitbosch
alexandru.ionescu