This patch adds the selected task information to the cache file info so that we create a new base try run instead of using a cached one. The cache is also converted to use a list of base pushes that each may have unique tasks that were run.
Differential Revision: https://phabricator.services.mozilla.com/D177298
This patch fixes an issue where the extra_args aren't reset between the base, and new runs. It also adds an additional failure when unknown arguments are provided.
Differential Revision: https://phabricator.services.mozilla.com/D177820
This patch adds documentation to the BasePerfComparator, and also adds some documentation for Comparators in general to the mach-try-perf Firefox source docs.
Differential Revision: https://phabricator.services.mozilla.com/D177760
This patch adds the ability to run custom comparisons through "comparators". The code is modified to create a BasePerfComparator that provides the existing default behaviour. A new method, and command-line arguments are added to accept either a path to a custom comparator, or a name of a builtin comparator.
To add new builtin comparators, they must be added to the perfcomparators.py file with an `@comparator` decorator. It must also subclass the BasePerfComparator. With this, a BenchmarkComparator is added that lets us pass custom options to raptor through the PERF_FLAGS environment variable (see --extra-args). To do this, some code needed to be moved around such as where the try_config is created. This lets us reset the configuration between the base, and new revision if needed.
The BenchmarkComparator lets us pass PR links as arguments, or the actual benchmark repo, revision, and branch. The PR link gets parsed into the required info using Github API requests. Note that links to direct commits do not work at the moment as the API doesn't provide the branch information for those kinds of requests.
Depends on D177030
Differential Revision: https://phabricator.services.mozilla.com/D177031
We now have windows custom chromium-as-release available in-tree for windows 64 platform.
This patch makes those tasks available for selection via mach try perf.
Differential Revision: https://phabricator.services.mozilla.com/D177671
Modify the selector so that the new `chromiums-as-release`/`custom-car` tasks are properly selected when a user submits a job. Previously custom-car was going through on firefox only selections. This patch should fix this as well as add a new --custom-car flag for selection.
Differential Revision: https://phabricator.services.mozilla.com/D177073
This patch moves some code to properly handle the max number of tasks better, and considers the --rebuild setting when we check if there are too many tasks selected.
Differential Revision: https://phabricator.services.mozilla.com/D174261
This patch adds a method to inject the perftest change-detector task into mach try perf try runs. To do this, we need to flip the ordering of the try runs to do the base revision first, and the one with the changes (new) afterwards. This allows us to link a change-detector task in the new try run, with the base try run by passing a revision through the try-task-config.
The task can be triggered using --detect-changes. Once a user pushes to try, and after all of the performance tests finish in that push, the change-detector will start running, to generate a comparison. An artifact is produced with the results (revisions, and metrics regressed), and an exception is raise when a performance change is detected even if it's an improvement.
Differential Revision: https://phabricator.services.mozilla.com/D172284
This patch allows mobile developers to upload custom APKs for testing through a commit. This allows them to run our performance tests by building locally, and then uploading to CI to run tests there.
The `./mach try perf` command is modified to make this simpler. It accepts either an environment variable, or a path to an APK, and copies it in-tree. After adding it to hg, the command stops running and asks the user to commit the changes. From there the user re-runs the `./mach try perf` command to select the appropriate tests.
Using --browsertime-upload-apk, users can use a custom APK for browsertime tests, and using --mozperftest-upload-apk, users can use a custom APK in mozperftest tests. The reason it's done this way is that we don't have common areas between the two frameworks. The methods are the same in both cases, i.e. for a fenix test, a fenix APK needs to be uploaded.
Differential Revision: https://phabricator.services.mozilla.com/D172435
This patch allows mobile developers to upload custom APKs for testing through a commit. This allows them to run our performance tests by building locally, and then uploading to CI to run tests there.
The `./mach try perf` command is modified to make this simpler. It accepts either an environment variable, or a path to an APK, and copies it in-tree. After adding it to hg, the command stops running and asks the user to commit the changes. From there the user re-runs the `./mach try perf` command to select the appropriate tests.
Using --browsertime-upload-apk, users can use a custom APK for browsertime tests, and using --mozperftest-upload-apk, users can use a custom APK in mozperftest tests. The reason it's done this way is that we don't have common areas between the two frameworks. The methods are the same in both cases, i.e. for a fenix test, a fenix APK needs to be uploaded.
Differential Revision: https://phabricator.services.mozilla.com/D172435
Currently, ./mach try ... --artifact only affects non-shippable (aka PGO) builds. This is fine for many use cases, but there are a number of tasks that depend exclusively on shippable builds, and it's when testing those types of tasks it usually doesn't matter if you have a full build, or whether or not it's PGO. --disable-pgo partly does this: it disables the profile run and instrumented build parts of shippable builds. When combined with --artifact, you end up with a single, artifact build - exactly what we want!
We should make this the default behaviour for --artifact. It's very surprising behaviour to pass --artifact and end up with a full, 2 stage build because you happen to be testing something that needs a shippable build.
Differential Revision: https://phabricator.services.mozilla.com/D169640
Currently, ./mach try ... --artifact only affects non-shippable (aka PGO) builds. This is fine for many use cases, but there are a number of tasks that depend exclusively on shippable builds, and it's when testing those types of tasks it usually doesn't matter if you have a full build, or whether or not it's PGO. --disable-pgo partly does this: it disables the profile run and instrumented build parts of shippable builds. When combined with --artifact, you end up with a single, artifact build - exactly what we want!
We should make this the default behaviour for --artifact. It's very surprising behaviour to pass --artifact and end up with a full, 2 stage build because you happen to be testing something that needs a shippable build.
Differential Revision: https://phabricator.services.mozilla.com/D169640
fzf doesn't quote the preview command arguments, so if you have
windows-style paths given to a unix-style shell, the slashes are
completely gone.
This happens on multiple systems, both with MozillaBuild and external
MSYS2 shell.
fzf respects the SHELL environment variable, so to fix it, execute the
command using cmd.exe on Windows, via $COMSPEC.
Depends on D169264
Differential Revision: https://phabricator.services.mozilla.com/D169265
This is _mostly_ boring, standard stuff. The most notable part is the refactor of the test transforms to process `variant` as early as possible. This is needed to key `suite` by variant (and is likely going to be helpful in other places as well, eg: we can now key all `mozharness` keys on variant). See the previous patch for why we need a separate xpcshell suite for this.
I don't know if I'm adding the runtime manifest correctly, or how those times are supposed to be generated/updated (the commit log seems to suggest that it's broken right now?) - let me know if I should do something differently here.
Note: I'm still working through a few more tests that need to be disabled, so this isn't ready to be landed yet, but I don't anticipate any changes to this patch.
Differential Revision: https://phabricator.services.mozilla.com/D164230
Greg Mierzwinski
Mark Banner
Kash Shampur
Paul Adenot
Myeongjun Go
Gijs Kruitbosch
Erich Gubler
Nicolas Silva
andrej
Julian Descottes
Iulian Moraru
Andrew Halberstadt
Cristian Tuns
Sandor Molnar
Ben Hearsum
Cosmin Sabou
Emilio Cobos Álvarez
alwu
Ben Hearsum
Marco Castelluccio