This patch adds a transformer that is capable of extracting times/entries or durations for logcat logs. Here, we also implement the ability for users to pass options to the transformers so that they can be more generic (through transformer-options). The logcat transformer makes use of this, and also shows how a custom function can be used within transformers. A new test file is added for the transformer, as well as a test in test_perfherder.py so we can test the full pipeline.
Differential Revision: https://phabricator.services.mozilla.com/D77094
This patch surfaces the timeout value for ADBDevice.
It also adds the ability to run a single test with
mach perftest-test and a new --skip-lint option
to skip black/flake8
Differential Revision: https://phabricator.services.mozilla.com/D76791
This patch fixes a bug where a PosixPath is attempted to be used rather than a string path. There's also a regression in the coverage test run in `./mach perftest-test` on some platforms that is fixed by this patch (the attrs module being used by pytest is too old, so we update during setup). Lastly, a regression test is added to the browsertime tests to ensure that the command only contains knowingly compatible types.
Depends on D76713
Differential Revision: https://phabricator.services.mozilla.com/D76599
This patch adds a new --perfherder-app that can be used to specify the shorthand name of the application being tested. The choices here are limited to what the perfherder schema can accept. We also add these settings to the existing taskcluster tasks to split the perfherder data. A couple tests are also added for this new flag.
Differential Revision: https://phabricator.services.mozilla.com/D76010
This patch adds a `test_name` field to the test metadata that can be used to change the name of the test when output in the perfherder format. It also implements this for the app-link tests.
Differential Revision: https://phabricator.services.mozilla.com/D75544
This patch is adding an option to push a perftest run in the CI.
It's based on :
- sparse profiles
- push_to_try
- options passed through try_task_config.json
Differential Revision: https://phabricator.services.mozilla.com/D74115
This patch enables perfherder data validation with jsonschema and also adds some extra tests for the Perfherder layer.
Differential Revision: https://phabricator.services.mozilla.com/D74877
Adds support for all variations of fenix (app name and activity passed by argument to mach perfest)
Fixes bug in mach perftest argument passing (android-activity was not being set)
Differential Revision: https://phabricator.services.mozilla.com/D73334
We need to set up the notebook deps.
The train of patch re-activated some
tests that required the dependencies used
by the notebook, which were not added in the
path.
Differential Revision: https://phabricator.services.mozilla.com/D73936
This patch implements the new intermediate results standard and adds the mechanisms required to handle it. Results validation is done with jsonschema and some manual validation (because of some unfortunate issues with jsonschema) and some tests were implemented to ensure that we fail/pass where expected. The metrics modules were modified to handle multiple suites.
One thing that is disabled in this patch is the subtest/single-metric specifications through the "results" field. We'll do one thing at a time here and we also have no use for subtests yet (although we definitely will).
Differential Revision: https://phabricator.services.mozilla.com/D72067
This patch adds support for tests metadata. A test script parser is added as
well as a new "doc" flavor that can be used to display the script info in the
command line. This parser will be the basis for building automated docs and
scripts verifications if we want to do this.
Differential Revision: https://phabricator.services.mozilla.com/D72800
We're back to an explicit --android option and --android-app-name, the layer won't use the --browser-binary option as an implicit
fallback anymore. This is cleaner.
Differential Revision: https://phabricator.services.mozilla.com/D72031
xuanqi xu
Tarek Ziadé
Tom Prince
Narcis Beleuzu
Gregory Mierzwinski
Yue Zhang
Andrew Creskey