For now, this just:
- updates the TreeHerder symbol
- includes the `--browsertime` extra option
The actual test jobs will react to the `--browsertime` extra option
and add dependencies, etc, as appropriate.
Differential Revision: https://phabricator.services.mozilla.com/D41605
--HG--
extra : moz-landing-system : lando
The current behaviour of resolve_tests_by suite is to look for exact
matches between the passed in path and the paths associated with each
test. But that doesn't work for cases where you want to path in an
ancestor path and run all the tests under that path.
This changeset makes resolve_tests_by_suite return a list of the input
paths that prefix-match one or more tests in a given suite, sorted by
suite. It is assumed that the harness itself will be able to handle
such inputs even when they don't correspond to a directory that
actualy contains tests.
Differential Revision: https://phabricator.services.mozilla.com/D42476
--HG--
extra : moz-landing-system : lando
Often times, PGO builds aren't required for testing things (in particular,
testing release automation). However, at least when testing release automation,
we do need to use the shippable build type.
Add an option to mach try that will disable using the 3-tier PGO jobs.
Differential Revision: https://phabricator.services.mozilla.com/D36365
--HG--
extra : moz-landing-system : lando
Templates invoke the `morph` logic, which is somewhat confusing and inflexible.
Update the machinery to support setting other `try_task_config` values.
Differential Revision: https://phabricator.services.mozilla.com/D36364
--HG--
extra : moz-landing-system : lando
Factor out the logic for calculating `try_task_config` from `push_to_try`,
so it can be called only for those selectors that need it.
Differential Revision: https://phabricator.services.mozilla.com/D36363
--HG--
extra : moz-landing-system : lando
Changes:
- require `--full` keyword for `./mach try fuzzy` in order to schedule android-hw jobs to hopefully reduce the backlog
Differential Revision: https://phabricator.services.mozilla.com/D33834
--HG--
extra : moz-landing-system : lando
- sm-shell: Selects shell only test cases that shouldn't require a full browser build.
- sm-all: Selects test cases that may require a full browser build.
Differential Revision: https://phabricator.services.mozilla.com/D28994
--HG--
extra : moz-landing-system : lando
Bug 1484243 removed the --git argument and made the VCS auto-detected,
so we should no longer suggest passing in that argument in error
messages.
Differential Revision: https://phabricator.services.mozilla.com/D30257
--HG--
extra : moz-landing-system : lando
Since we need to generate the full_task_set as a prereq to the target_task_set,
and since getting from full_task_set -> target_task_set is trivial.. we might
as well save both computed sets to the cache while we have them. This means users
that run:
$ ./mach try fuzzy
and then run:
$ ./mach try fuzzy --full
(or vice versa)
Will only incur task generation once. It also means that the 'watchman' trigger
will cache both taskgraphs.
Differential Revision: https://phabricator.services.mozilla.com/D29557
--HG--
extra : moz-landing-system : lando
This adds a 'watchman.json' file to /tools/tryselect and some documentation on
how to use it. Tl;dr, install watchman and then:
$ cd path/to/gecko
$ watchman -j < tools/tryselect/watchman.json
Differential Revision: https://phabricator.services.mozilla.com/D28771
--HG--
extra : moz-landing-system : lando
It turns out there are several places where the change to suite 'jittest-chunked'
causes problem. I am abandoning that approach.
Desktop uses this trick, and this returns android '-chunked' handling to a state
similar to what it was before I started messing around!
Differential Revision: https://phabricator.services.mozilla.com/D28897
--HG--
extra : moz-landing-system : lando
This skips a number of talos jobs that are unlikely to be affected
by due a browser-chrome specific change
Run with: `./mach try fuzzy --preset perf-chrome`
Differential Revision: https://phabricator.services.mozilla.com/D27911
--HG--
extra : moz-landing-system : lando
This officially makes 'moztest.resolve' the source of truth when it comes to
suite names. It aligns that file with the names used in both the
desktop_unittest and android_emulator_unittest scripts.
Differential Revision: https://phabricator.services.mozilla.com/D27555
--HG--
extra : moz-landing-system : lando
Currently we have the concept of a "suite" and a "flavour" in our task
configuration. Typically, the "suite" refers to the high-level test harness
like "mochitest" or "reftest", whereas the flavour is more specific, e.g
"browser-chrome-instrumentation" or "crashtest". However the line between suite
and flavour is not applied with any semblance of consistency which results in
inconsistent naming throughout the tree.
This patch gets rid of the concept of "flavours" entirely (at least when it
comes to task configuration). A suite is a type of test run, for example:
- mochitest-plain
- mochitest-devtools-chrome
- mochitest-browser-chrome-instrumentation
- jsreftest
- reftest
- firefox-ui-functional-remote
etc
There is no confusion here between suites and flavours because flavours don't
exist. However, there are a couple of places where we *do* need to know what
"test harness" is used to run a suite. These cases are:
1. For SCHEDULES moz.build rules
2. For the desktop_unittest.py mozharness script which takes arguments like
--mochitest-suite=browser (this is not a compelling use of this information
and should be refactored to work more like the android_emulator_unittest.py
script)
So to get this information, this patch introduces a new concept of a "category"
which is the overall "test harness" that runs the suite. For many suites, the
"category" is identical to the suite name. Unlike flavours, "categories" have
no bearing on how we call or refer to the suite.
Differential Revision: https://phabricator.services.mozilla.com/D27554
--HG--
extra : moz-landing-system : lando
This was regressed by bug 1544816 but the test never ran on the push that regressed.
This patch also updates the 'files-changed' for the tryselect task.
Differential Revision: https://phabricator.services.mozilla.com/D28386
--HG--
extra : moz-landing-system : lando
Changes:
- added windows10-aarch64 to the filter for fuzzy, to require `--full` in order to trigger jobs
- return False for any test tasks that contain windows10-aarch64 to prevent users using old try syntax from overwhelming the limited number of hardware
Differential Revision: https://phabricator.services.mozilla.com/D27590
--HG--
extra : moz-landing-system : lando
Since e10s is the default configuration, we shouldn't explicitly mark things
with the "-e10s" suffix. Instead we should mark things that *don't* run with
'e10s. This patch removes '-e10s' from all treeherder group symbols and task
labels, adds the "-1proc" suffix to tasks that are non-e10s.
Differential Revision: https://phabricator.services.mozilla.com/D25958
--HG--
extra : moz-landing-system : lando
This helps run the tests locally if fzf is normally installed in
$HOME/.mozbuild. Since the tests set MOZBUILD_STATE_PATH to a temporary
directory, fzf_bootstrap() can't find fzf in the HOME location unless it
is added to PATH.
Differential Revision: https://phabricator.services.mozilla.com/D27194
--HG--
extra : moz-landing-system : lando
Add Android 7.0 gtests, opt and debug, running against the geckoview
TestRunnerActivity.
Differential Revision: https://phabricator.services.mozilla.com/D27016
--HG--
extra : moz-landing-system : lando
There are a few windows/mac-only suites that were missed since we were only
reading the linux variant.
Depends on D25401
Differential Revision: https://phabricator.services.mozilla.com/D25432
--HG--
extra : moz-landing-system : lando
While running presets + intersection queries works with 'mach try fuzzy
--preset <foo>', it is still broken with 'mach try --preset <foo>'.
Differential Revision: https://phabricator.services.mozilla.com/D25298
--HG--
extra : moz-landing-system : lando
Some of these were working with the '<flavor>-<subsuite>' mechanism that was
previously being used, but better to be explicit wherever possible.
Depends on D25077
Differential Revision: https://phabricator.services.mozilla.com/D25078
--HG--
extra : moz-landing-system : lando
Mozharness appends -chunked/-coverage to some suites, but the build system/test
resolver don't have any concept of these things. We need to normalize these out
for the purposes of MOZHARNESS_TEST_PATHS.
Differential Revision: https://phabricator.services.mozilla.com/D25015
--HG--
extra : moz-landing-system : lando
It turns out bug 1489100 regressed the ability to specify test paths for most
suites by naively assuming that mozharness uses suite names that look like:
<flavor>-<subsuite>
In reality, there is no consistency to how suite names are generated. This test
does a few things:
1) Patches the moztest.TestResolver to return a list of all possible
suites/subsuites (assuming the lists in moztest.resolve are up to date).
2) Finds all the suites defined in the mozharness configs (e.g
linux_unittest.py), and uses these are parametrized inputs.
3) Checks that for each test suite,
DesktopUnittest._get_mozharness_test_paths() returns something.
I've marked all of the test suites that currently fail as expected. This way I
have a good sense of what needs to be fixed, and can validate my changes as I
move through the list.
Differential Revision: https://phabricator.services.mozilla.com/D25014
--HG--
extra : moz-landing-system : lando
Nick Alexander
James Graham
championshuttler
Edwin Gao
Nicolas Silva
Sylvestre Ledru
Simon Giesecke
Tom Prince
Victor Porof
Adam Gashlin
Alexandre Poirot
Brian Grinstead
Matthew Gaudet
Julian Descottes
Mike Shal
Matthew Noorenberghe
Andrew Halberstadt
Joel Maher
Geoff Brown
neha
Justin Wood
Cosmin Sabou