This patch integrates mozpower into raptor desktop testing. It can be used for MacOS power testing by supplying the `--power-test` command at the command line.
Some changes to how the `--power-test` command is parsed are also made so that we don't check for the `--host` argument when it is supplied. Now, it is only checked when `--app` is an android browser, in the near future this `--host` argument will no longer be needed for power testing on android.
Another change in this patch is the addition of the `self.artifact_dir` property which returns the current directory that should be used to output the artifacts.
Power usage data is output into this artifact directory in a 'power-measurements' folder and this data is zipped when we are running in CI.
Differential Revision: https://phabricator.services.mozilla.com/D42014
--HG--
extra : moz-landing-system : lando
This patch adds support for using `./mach raptor` instead of `./mach raptor-test`. The old version is left for backwards compatibility.
Differential Revision: https://phabricator.services.mozilla.com/D43051
--HG--
extra : moz-landing-system : lando
This patch fixes an issue where if browser_cycle does not equal 1 for the first entry in the raptor results, then it will attempt to access combined_suites with a key entry which would only get created when we had an entry with browser_cycle equal to 1. With this change, we use the suite name to determine when we need to initialize the combined_suites entry. This issue occurs when the first cycle does not produce results.
Differential Revision: https://phabricator.services.mozilla.com/D43060
--HG--
extra : moz-landing-system : lando
The alert_on values from the test INI were not being passed into the test settings json via manifest.py, this patch will fix this.
Differential Revision: https://phabricator.services.mozilla.com/D43049
--HG--
extra : moz-landing-system : lando
This requires generalizing the existing flags from paths to possibly
being paths. The `os.path.expandvars` allows to refer to
`MOZ_FETCHES_DIR` symbolically in the build scripts (which aren't
invoked by the shell before execution, leaving us to expand shell
variables).
Differential Revision: https://phabricator.services.mozilla.com/D41604
--HG--
extra : moz-landing-system : lando
This picks up various improvements, especially to how configurations are handled and some new rules.
Differential Revision: https://phabricator.services.mozilla.com/D37271
--HG--
extra : moz-landing-system : lando
This patch adds two new test tasks for each of the 4 Android browsers running speedometer, idle, or idle-background. These tests gather (i) cpu+memory usage, or (ii) cpu+memory+power usage information. The power tests are left as they are for now to determine if there is an impact to our measurements if we run all resource-usage tests at once. Furthermore, the cron job for android power tests was also modified to prevent it from picking up these tasks.
The cpu usage functons needed to be fixed here because they were not working on MotoG5/Android7 for two reasons: (i) device version was not being detected properly, and (ii) ADB shell commands can't use the pipe functionality (`|`). Tests needed to be modified to compensate for these changes as well.
Differential Revision: https://phabricator.services.mozilla.com/D42679
--HG--
extra : moz-landing-system : lando
Based on the conclusion from the bug mentioned above, we will run youtube playback always muted on desktop.
Differential Revision: https://phabricator.services.mozilla.com/D41555
--HG--
extra : moz-landing-system : lando
This patch disables adaptive brightness on any android device that runs power tests. It is not restored because it should always be disabled.
Differential Revision: https://phabricator.services.mozilla.com/D41444
--HG--
extra : moz-landing-system : lando
Changed the required yml, ini, json, js and html files to add Jetstream2 benchmark test to Raptor.
The changes on the javascript files are on github.
Differential Revision: https://phabricator.services.mozilla.com/D35418
--HG--
extra : moz-landing-system : lando
This patch fixes four things at once:
(1) Consolidate supporting raptor measurements into one PERFHERDER_DATA output per type. For example, with this change, all power data will be grouped into one PERFHERDER_DATA output.
(2) Output perfherder-data-<DATA_TYPE>.json for each supporting measurement instead of overwriting perfherder-data.json which contains the regular raptor test results.
(3) Take an average of the supporting measurements when particular unit's are specified. In this case, the '%' unit makes us take the average instead of the sum of the measurements.
(4) Remove the redundant test name entry that prefixes all power subtest entries.
Differential Revision: https://phabricator.services.mozilla.com/D40667
--HG--
extra : moz-landing-system : lando
Add functionality for being able to send extra parameters for the test_url query of a test, directly from a taskcluster config.
Also, the PR adds logic in the `setup-raptor` taskgraph transform for dynamically changing the subset of youtube-playback tests based on the platform and project
Differential Revision: https://phabricator.services.mozilla.com/D39006
--HG--
extra : moz-landing-system : lando
The goal is to configure browsertime in Raptor in two ways:
1) locally, just like `mach browsertime` does;
2) in automation, at taskgraph creation time, using fetches and
mozharness suite artifacts (for geckodriver).
It's possible for this to be done using mozharness config settings but
using command line options is more explicit and more likely to be easy
to remove later if and when we transition to a browsertime-specific
mozharness script.
Differential Revision: https://phabricator.services.mozilla.com/D38776
--HG--
extra : moz-landing-system : lando
This mechanical transform lays the foundation for generalizing the
underlying test runner.
Differential Revision: https://phabricator.services.mozilla.com/D37132
--HG--
extra : moz-landing-system : lando
This mechanical transform lays the foundation for generalizing the
underlying test runner.
Differential Revision: https://phabricator.services.mozilla.com/D37132
--HG--
extra : moz-landing-system : lando
This mechanical transform lays the foundation for generalizing the
underlying test runner.
Differential Revision: https://phabricator.services.mozilla.com/D37132
--HG--
extra : moz-landing-system : lando
Bug 1547135 added functionality to configure specific Android devices
for performance tests. That code landed as part of device setup in
`testing/raptor/raptor.py` but there is nothing Raptor-specific
(indeed, nothing harness-specific) about this configuration.
Rather than lift it to `mozdevice`, this commit moves it into a
`performance_tuning` module. That's enough to slim down `raptor.py`.
Differential Revision: https://phabricator.services.mozilla.com/D37128
--HG--
extra : moz-landing-system : lando
Handle the exception ADBProcessError raised by
thermal_zone0 = self.device.shell_output('cat sys/class/thermal/thermal_zone0/temp')
on the emulator
Differential Revision: https://phabricator.services.mozilla.com/D38859
--HG--
extra : moz-landing-system : lando
Added the crash method for RaptorLogger class,
which will call structuredlog.py crash method.
Differential Revision: https://phabricator.services.mozilla.com/D38055
--HG--
extra : moz-landing-system : lando
This patch adds an additional data output to android power tests. This data is the power usage of the test calculated as a percentage increase relative to the OS baseline. test_power.py needed to be changed to accommodate these changes as well.
Differential Revision: https://phabricator.services.mozilla.com/D37462
--HG--
extra : moz-landing-system : lando
Changed the required yml, ini, json, js and html files to migrate ARES6 benchmark test to Raptor.
Differential Revision: https://phabricator.services.mozilla.com/D34178
--HG--
extra : moz-landing-system : lando
This avoids error messages about missing minidump directories when an unrelated failure aborts a run before launching the test app.
Differential Revision: https://phabricator.services.mozilla.com/D36925
--HG--
extra : moz-landing-system : lando
This patch adds three unit tests (in one file) to the set of the raptor tests. The tests include (i) an Android 8 power parsing test, (ii) an android 7 power parsing test, and (iii) a test for android OS baseline parsing (tested using android 8 data).
Differential Revision: https://phabricator.services.mozilla.com/D36926
--HG--
extra : moz-landing-system : lando
Florin Strugariu
Rob Wood
Stephen Donner
Gregory Mierzwinski
Marian Raiciof
Nick Alexander
Mark Banner
krubin
chrmod
Andrew Creskey
Alexandru Irimovici
Narcis Beleuzu
Gijs Kruitbosch
alexandru.ionescu
Geoff Brown
Henrik Skupin