In the past, the 'backstop' optimization was applied to tasks by default across
all projects, even though it only really made sense on autoland. To choose what
would happen on non-autoland branches, we invented this 'remove_on_projects'
concept.
These days, we only apply the backstop optimization in the first place for
autoland. So 'remove_on_projects' is no longer necessary.
Depends on D88149
Differential Revision: https://phabricator.services.mozilla.com/D88150
This removes the last uses of the 'push-interval-10' and 'push-interval-20' strategies.
They are being removed because they are dangerous in that its easy to accidentally not run
tasks when they should.
Instead, task authors should decide whether they want their tasks to run on
"backstop" pushes (run everything) or "expanded" pushes (run more than usual,
but still not as much as a backstop). Note that using "expanded" means the task
will *also* run on backstop pushes. It'll just additionally run on "expanded"
pushes.
In practice 'backstop' pushes will be every 20th push and 'expanded' pushes
will be every 10th push. Though this may vary due to the time component in
backstops.
Differential Revision: https://phabricator.services.mozilla.com/D89503
This is a nomenclature change + refactoring. Now there is only a single
"backstop" push. Which is currently set to every 20th push on autoland (or
every push on non-autoland branches).
Now there is also a concept of an "expanded" push. These are pushes that run
more stuff than usual, but not as much as a backstop normally would. These are
currently set to run at half the interval of a backstop.
Concretely, here are the strategy changes:
* Renamed 'backstop' -> 'skip-unless-backstop'
* Introduced the 'skip-unless-expanded' strategy which has 'skip-unless-backstop' baked in
* Merged 'optimized-backstop' into the 'test' strategy (with expanded comments)
* Baked both 'skip-unless-expanded' and 'skip-unless-backstop' into the 'test' strategy
Differential Revision: https://phabricator.services.mozilla.com/D89501
It turns out that 'Not' is needed to negate "backstops". E.g, we normally
we want to use a pattern like so:
All("skip-unless-backstop", "test")
Since 'skip-unless-backstop' returns False on backstop pushes, it disables
the test strategy there.
However, suppose we wanted to run a special optimization, *only* on backstop
pushes. I.e, the opposite of the above example. Then we need to use:
All(Not("skip-unless-backstop"), "test-backstop")
Depends on D89500
Differential Revision: https://phabricator.services.mozilla.com/D89734
The intent of a "backstop" push, is to run everything so we can be absolutely sure that
the push in question does not cause any regressions.
Previously, backstops were thought to be only something that ran on autoland.
This was because the other branches *already* ran everything so the concept of
a "backstop" didn't make much sense.
But going by the above definition, it would make more sense to say that *every*
non autoland (or try) push is a backstop. Since the intent there is to run
everything to avoid regressions.
This change will allow us to simplify our optimization algorithms.
Differential Revision: https://phabricator.services.mozilla.com/D89499
The script uses the `cd $FOO && pwd` trick to fix the path style on Windows, but currently this happens before the `mkdir`, so we get an incorrect result:
```
[task 2020-09-11T03:38:59.656Z] ++ cd Z:/task_1599794667/public/build
[task 2020-09-11T03:38:59.656Z] ./src/taskcluster/scripts/builder/build-sm.sh: line 10: cd: Z:/task_1599794667/public/build: No such file or directory
[task 2020-09-11T03:38:59.656Z] ++ pwd
[task 2020-09-11T03:38:59.657Z] + export MOZ_UPLOAD_DIR=/z/task_1599794667
```
Differential Revision: https://phabricator.services.mozilla.com/D89925
This fixes a bug that was introduced in recent optimizations where
upload-symbols may not run for shipped release builds. We can add it as
a dependency for this, or some other per-platform task that runs for
releases. As long as it runs before we ship this should be fine.
Differential Revision: https://phabricator.services.mozilla.com/D89494
Corresponding debug and shippable tests already run as tier 1, but the plain
opt tests (which run on autoland and try) currently run as tier 2, apparently by
accident.
Differential Revision: https://phabricator.services.mozilla.com/D89255
Pipenv is heavy weight and overkill for the purposes it is being used. We'd like to remove it from the tree and |mach python-test| was one of the last remanining use cases.
Remove the `--python` command-line argument as a result. Users who wish to run unit tests with Python 2 can do `MACH_PY2=1 ./mach python-test ...` or `python2 ./mach python-test ...`.
Also update a few unit tests that would break otherwise in the presence of this change.
There were a couple lines in the `setup.py` for `mozlog` that were problematic for tests and was resulting in errors due to the `mozlog` plugin being loaded by `pytest` more than once. We just delete those lines and bump up the major version number of the package to fix it.
Differential Revision: https://phabricator.services.mozilla.com/D88296
We want to try to align 'push-interval' tasks to the 'backstop'. This way
we have greater confidence in our backstop pushes, and it will allow us to
simplify a lot of our backstop logic.
Depends on D89055
Differential Revision: https://phabricator.services.mozilla.com/D89056
Each file represents a new taskgraph generation in the tests. But now that these exist, we
can add new assertions to the existing files without worry.
Depends on D89054
Differential Revision: https://phabricator.services.mozilla.com/D89055
Under this model, each test file should correspond to a static set of parameters. Each
set of parameters will only ever generate the taskgraph a single time (so adding a new
file will have a perf hit, but adding new test functions to an existing file will not).
Differential Revision: https://phabricator.services.mozilla.com/D89054
The new Cranelift compiler for WebAssembly makes jittest run a bit slower, so
raise the timeout value for those tests, until performance enhancements make it
run faster again.
Differential Revision: https://phabricator.services.mozilla.com/D88973
Backstop pushes have a time component so that we can guarantee they run at least every N
hours. Prior, they measured the time since the last push. This means that as long as a new
push comes in at least once every 4 hours, the backstop would keep being pushed out.
This means we could theoretically go 80 hours without a backstop push, which makes finding
merge candidates tricky.
This change ensures we measure the 4 hours as time since last backstop, as opposed to time
since last push.
Differential Revision: https://phabricator.services.mozilla.com/D88913
Sometimes we want to conditionally add index routes to the decision task
based on a parameter. E.g, a route that tracks all the backstop pushes.
This adds a new index task specifically for the decision task, which allows
to defined new index routes within the decision task itself.
Differential Revision: https://phabricator.services.mozilla.com/D88911
I found myself needing to test a push with 'backstop=True' on try. Though, I
couldn't set it because the call to 'is_backstop' happens after the
'try_task_config' parameters are added in. This adds a simple check to handle
that case.
Differential Revision: https://phabricator.services.mozilla.com/D88910
We had thought ReferenceBrowser tests would be useful in isolating Fenix startup regressions.
However the startup of the Reference Browser is not directly comparable to Fenix and so we have not used these tests.
Differential Revision: https://phabricator.services.mozilla.com/D88666
Dorel Luca
Dzmitry Malyshau
Ben Hearsum
Andrew Halberstadt
Andrew Halberstadt
David Major
Doug Thayer
Joel Maher
Sylvestre Ledru
Narcis Beleuzu
Tom Ritter
Cristina Coroiu
stransky
Andreea Pavel
Geoff Brown
Gregory Mierzwinski
Mike Hommey
Mihai Tabara
Ricky Stewart
Benjamin Bouvier
Jeff Gilbert
Andrew Creskey