This adds a multi-e10s variant for geckoview-junit tests. With bug 1622944
resolved, the test suite passes, so we allow this variant to be tier-1.
Differential Revision: https://phabricator.services.mozilla.com/D66676
- changed test URL to match the dev server
- changed output.py in several places to fix new test names, dict keys, to cover all tests
- added amazonaws.com to manifest.json file to fix the loading issue for benchmark.js file
- added all raptor tests
- changed the constants for measure and alert_on
Differential Revision: https://phabricator.services.mozilla.com/D62546
The default way to split the 'arg' parameter for CompositeStrategies, is to
duplicate it across all substrategies. By setting 'split_arg=tuple', we instead
break the arg up so the first index goes to the first substrategy, the second
index goes to the second substrategy, etc.
This means that the length of the 'test' arg must be at least as long as the
number of substrategies.
Differential Revision: https://phabricator.services.mozilla.com/D72464
This patch prevents speedometer tests from running on all android builds and also prevents chrome tests from running there.
Differential Revision: https://phabricator.services.mozilla.com/D72727
This patch changes how Google Chrome for Android is deployed. Rather than relying on automatic updates, we will use tooltool to download the APK and install it ourselves. Some changes were done in taskcluster to remove a hack that was put in place to disable interal tooltool downloads (the issue is resolved now).
A tooltool manifest is added for this, and to keep ourselves organized, all manifests (including the playback ones) are moved into a folder called `tooltool-manifests`.
Differential Revision: https://phabricator.services.mozilla.com/D72198
The custom retrigger actions work well on linux and android-em, but fail
on windows, osx, and android-hw. At least part of the problem seems to be
the worker implementation, but I am not entirely clear on what goes wrong.
It looks like I won't have much more time for retrigger improvements in the
near future, so I'd prefer to "turn off" the actions on tasks known to fail.
I found helpful examples for the 'context' parameter in
https://searchfox.org/mozilla-central/source/taskcluster/docs/actions.rst
Differential Revision: https://phabricator.services.mozilla.com/D72233
This adds a parameter that will cause a task to sum all the confidence
thresholds of the relative manifests it contains to gather a larger overall
task confidence.
This also adds a new strategy + shadow-scheduler to go along with it.
Differential Revision: https://phabricator.services.mozilla.com/D71314
This patch adds the new live site tests as Raptor-Browsertime tasks in CI. These will be scheduled to run through the general-perf-testing cron task on Monday/Wednesday/Friday.
Differential Revision: https://phabricator.services.mozilla.com/D69053
This change is beneficial for two reasons:
1) Improvement on the single responsibility principle
2) Platform filter isn't 'bugbug' specific. E.g, the 'relevant tests' optimizer
could also theoretically use this. Having it as a standalone optimizer allows
us to compose it with other strategies.
Differential Revision: https://phabricator.services.mozilla.com/D71210
This ensures we don't run every build with every push via ./mach try auto. It
introduces a new 'optimization-overrides' try_config that can be used to
replace optimizations. For now, there is no user interface to pass this in via
the 'mach try' command line.
Differential Revision: https://phabricator.services.mozilla.com/D68207
Changes:
Applies the `filter_tasks_by_blacklist` method to try syntax pushes as well.
- moved `TARGET_TASK_BLACKLIST`and `filter_tasks_by_blacklist` method to live in `taskcluster/taskgraph/target_tasks.py`.
- removed existing filters against `ccov, windows10-aarch64` and `android-hw` filters against try syntax pushes.
- update imports for `fuzzy` and `chooser` selectors to refer to the new location of `filter_tasks_by_blacklist` method.
The reason for moving the logic (again) from `tools/tryselect` to `taskcluster/` is due to the placement of `try_option_syntax` and `target_tasks` files and both of those files handle the processing of `mach try syntax` pushes.
Differential Revision: https://phabricator.services.mozilla.com/D71698
Various updates to the custom retrigger action so that, without any custom changes to
parameters, the retriggered task runs with the same parameters as the original task.
Several issues were found and corrected, notably:
- parameters like --allow-software-gl-layers were ignored
- MOZHARNESS_TEST_PATHS was ignored
- many parameter customizations in the desktop mozharness configs were ignored
- mochitest suite/subsuite/flavor selection was not always correct
- using repeat=1 by default meant that each test ran twice
Differential Revision: https://phabricator.services.mozilla.com/D70457
Keeping the same for the currently chosen strategy for try auto, since we
don't want to decrease its regression detection rate.
We also add a new shadow scheduler which uses the reduced set with a higher
confidence threshold.
Differential Revision: https://phabricator.services.mozilla.com/D71205
This patches fixes several problems found on Raptor and the condprof:
Raptor:
- Make sure the conditioned profile dir is removed after
it's been used, not before.
- Adds the --project option to raptor so we know if we're on try
autoland or mozilla-central.
- Both Fennec and Fenix are deactivated for now
- Use the allow-downgrade flag to be flexible on build ids (the next step will be bug 1628666)
Conditioned profiles, curation of the profile prefs:
- Fully deactivates Normandy during Raptor tests (app.normandy.enabled)
- Removes any GFX blacklisting (gfx.blacklist.*)
- Removes any marionette pref
- Enforce extensions sideloading (extensions.startupScanScopes)
Differential Revision: https://phabricator.services.mozilla.com/D70518
We'll want some kind of backstop no matter what optimization algorithm we use.
We don't want to go too long without running any given task so we can find
regressions quickly and have a good merge candidate.
This pulls the logic that handles this out of the SETA strategy and into its
own strategy.
This will also make the SETA shadow scheduler more representative of what the
algorithm is doing.
Note in the future we may find ways to make this backstop more efficient (i.e
only run tasks that didn't run in the last 9 pushes for example).
Depends on D68621
Differential Revision: https://phabricator.services.mozilla.com/D68622
--HG--
extra : moz-landing-system : lando
This allows to nest strategies without having to register ever intermediate
composite strategy first. For example:
All(Any("skip-unless-schedules", "seta"), "backstop")
Prior to this patch, we'd need to register that 'Any' one first and then use it
in the 'All'.
Depends on D68620
Differential Revision: https://phabricator.services.mozilla.com/D68621
--HG--
extra : moz-landing-system : lando
I'd like to implement a 'backstop' strategy, such that it will prevent all other
optimizers from removing tasks under certain conditions (e.g every 10th push).
The nicest way to implement this seems to be an 'All' composite strategy
(similar to 'Either' which this patch renames to 'Any'). This means we could
do something like:
All("seta", "backstop")
which means we would only remove tasks if *all* substrategies say to remove
tasks.
Differential Revision: https://phabricator.services.mozilla.com/D68620
--HG--
extra : moz-landing-system : lando
We'll want some kind of backstop no matter what optimization algorithm we use.
We don't want to go too long without running any given task so we can find
regressions quickly and have a good merge candidate.
This pulls the logic that handles this out of the SETA strategy and into its
own strategy.
This will also make the SETA shadow scheduler more representative of what the
algorithm is doing.
Note in the future we may find ways to make this backstop more efficient (i.e
only run tasks that didn't run in the last 9 pushes for example).
Depends on D68621
Differential Revision: https://phabricator.services.mozilla.com/D68622
--HG--
extra : moz-landing-system : lando
This allows to nest strategies without having to register ever intermediate
composite strategy first. For example:
All(Any("skip-unless-schedules", "seta"), "backstop")
Prior to this patch, we'd need to register that 'Any' one first and then use it
in the 'All'.
Depends on D68620
Differential Revision: https://phabricator.services.mozilla.com/D68621
--HG--
extra : moz-landing-system : lando
I'd like to implement a 'backstop' strategy, such that it will prevent all other
optimizers from removing tasks under certain conditions (e.g every 10th push).
The nicest way to implement this seems to be an 'All' composite strategy
(similar to 'Either' which this patch renames to 'Any'). This means we could
do something like:
All("seta", "backstop")
which means we would only remove tasks if *all* substrategies say to remove
tasks.
Differential Revision: https://phabricator.services.mozilla.com/D68620
--HG--
extra : moz-landing-system : lando
Activates the conditioned profile by doing the following changes:
- make sure the conditioned profile dir is removed after
it's been used, not before.
- add the --project option to raptor so we know if we're on try
or mozilla-central.
- Both Fennec and Fenix are deactivated for now.
- Remove any gfx.blacklist.* prefs when using a conditioned profile
Differential Revision: https://phabricator.services.mozilla.com/D70518
--HG--
extra : moz-landing-system : lando
Make sure the conditioned profile dir is removed after
it's been used, not before. This patch also adds the
--project option to raptor so we know if we're on try
or mozilla-central. Both Fennec and Fenix are deactivated
for now.
Differential Revision: https://phabricator.services.mozilla.com/D70518
--HG--
extra : moz-landing-system : lando
Since we don't look at the changesets, there is no need for hgmo to generate
them for us.
Differential Revision: https://phabricator.services.mozilla.com/D70567
--HG--
extra : moz-landing-system : lando
Previously source-tests requiring a build had a "global" mapping of
platform to build type in kind.yml. But this made it confusing to
figure out how to add task-specific configuration. To simplify things,
make the configuration for the dependent platforms also go in the task
definition.
Differential Revision: https://phabricator.services.mozilla.com/D68636
--HG--
extra : moz-landing-system : lando
Aaron Klotz
Marian Raiciof
Andrew Halberstadt
Gregory Mierzwinski
Nick Thomas
Joel Maher
Geoff Brown
Bogdan Tara
Bob Clary
Simon Fraser
Arthur Iakab
Kershaw Chang
Edwin Takahashi
Tom Prince
Marco Castelluccio
Tarek Ziadé
Andreea Pavel
Mihai Alexandru Michis
Aki Sasaki
Csoregi Natalia
Coroiu Cristina
Rob Lemley
James Graham