These files were omitted from the original patch because reformatting them required some manual intervention in order to avoid breaking unit tests. Generally the `noqa` lines were already there and just needed to be moved from one line to another (due to the reformatting by `black`), but sometimes `black` saw fit to move a bunch of stuff all onto one line, requiring me to introduce new `noqa` lines.
Besides the autoformat by `black` and some manual fixups, this patch contains no other changes.
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94052
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
This walks the file system to read moz.build files, rather than following the DIRS
traversal. The latter is problematic because many moz.build files that happen to
define manifests could be excluded by the local build config.
Differential Revision: https://phabricator.services.mozilla.com/D94197
This was causing us to generate the build backend on 'tab' completion (since we
were importing the file to parse available options out of the ArgumentParser
defined therein).
Differential Revision: https://phabricator.services.mozilla.com/D92010
This can be run using:
$ ./mach try --preset builds
The build list can be filtered further using '-xq', e.g:
$ ./mach try --preset builds -xq "win"
Or it can be filtered further interactively:
$ ./mach try --preset builds -xi
Finally, tasks can be added to the build list:
$ ./mach try --preset builds -q "'windows 'mochitest"
This gives a T-style push. There is currently no way to both filter builds and
add additional tasks, it needs to be one or the other.
Differential Revision: https://phabricator.services.mozilla.com/D81352
Today we don't require that `mach` `CommandProvider`s subclass from any particular parent class and we're very lax about the requirements they must meet. While that's convenient in certain circumstances, it has some unfortunate implications for feature development.
Today the only requirements that we have for `CommandProvider`s are that they have an `__init__()` method that takes either 1 or 2 arguments, the second of which must be called `context` and is populated with the `mach` `CommandContext`. Again, while this flexibility is occasionally convenient, it is limiting. As we add features to `mach`, having a better idea what the shape of our `CommandProvider`s are and how we can instantiate them and use them is increasingly important, and this gives us additional control when having `mach` configure `CommandProvider`s based on data that is only available at the `mach` level. In particular, we plan to leverage this in bugs 985141 and 1654074.
Here we add validation to the `CommandProvider` decorator to ensure all classes inherit from `MachCommandBase`, update all `CommandProvider`s in-tree to inherit from `MachCommandBase`, and update source and test code accordingly.
Follow-up work: we now require (de facto) that the `context` be populated with a `topdir` attribute by the `populate_context_handler` function, since instantiating the `MachCommandBase` requires a `topdir` be provided. This is fine for now in the interest of keeping this patch reasonably sized, but some additional refactoring could make this cleaner.
Differential Revision: https://phabricator.services.mozilla.com/D86255
Try syntax users wishing to preserve their default selector can modify
~/.mozbuild/machrc (create it if it doesn't exist), and add:
[try]
default = syntax
Depends on D82983
Differential Revision: https://phabricator.services.mozilla.com/D82985
Since partials have started verifying signatures, the partial task has been failing in
`mach try scriptworker`. Since we are not concerned with the partial task itself,
split the tasks into two groups, so that it does not need to run.
Differential Revision: https://phabricator.services.mozilla.com/D83370
There is some remaining code in central originating from bug 1184405, which sought to associate source files with their "affected" test files. That ended up not panning out, and bug 1644228 removed a lot of that code, but left some remnants in the `Files` object which are still referenced in a couple different places. I'm deleting all of that code in `context.py` plus everything that references it for the following reasons:
1. Right now, `Files.{test_files,test_tags,test_flavors}` do get populated, but only ever with "default" values -- namely `moz.build` files that are above the files in question in the directory hierarchy. This is a heuristic that doesn't actually have anything to do with mapping source files to their corresponding test files, which is misleading.
2. Those attributes are accessed in two places. The first is in the `mach file-info dep-tests` command. This command isn't referenced anywhere else in tree and I don't have any evidence anyone ever uses it. Even if they do, I would claim that doing so is a mistake (because the results of the command aren't meaningful and are just populated by the "defaults" described above), and that person's workflow should be migrated to something else that *is* meaningful.
3. The second place where this metadata is accessed is in `testing/mozbase/moztest/moztest/resolve.py`; that method is invoked in `tools/tryselect/selectors/syntax.py`, but only if you pass `--detect-paths` to `mach try syntax`. This is [entirely broken](https://bugzilla.mozilla.org/show_bug.cgi?id=1614614), and even if we made an effort to fix it, it wouldn't do anything resembling what the documentation of `--detect-paths` suggests it would do (again, because the data isn't populated meaningfully). So I'm deleting the command line option entirely.
Differential Revision: https://phabricator.services.mozilla.com/D79711
When python2 interprets headers here, it will attempt to encode values into a string in a blanket fashion.
In this case, the `Accept` header is encoded into an invalid value, but it "works".
Python3 validates more strictly, and throws an error when an array is encountered as a value here. Formatting it
as a valid string instead resolves the problem.
Differential Revision: https://phabricator.services.mozilla.com/D77735
This patch is adding an option to push a perftest run in the CI.
It's based on :
- sparse profiles
- push_to_try
- options passed through try_task_config.json
Differential Revision: https://phabricator.services.mozilla.com/D74115
Mach doesn't know which tasks are part of a release - that's decided in the decision task. Instead of
showing a bogus estimation, we shouldn't show one at all.
Differential Revision: https://phabricator.services.mozilla.com/D74904
In some NT-specific code, a list of "items()" was being updated. In python 3, to modify the result
of "items()", you have to gain ownership first by explicitly converting it into a list().
Also resolves some flaky failures that were seen locally by marking test_presets.py as sequential.
Differential Revision: https://phabricator.services.mozilla.com/D74793
|./mach try| subcommands are now compatible with both python 2 and 3.
Hand-tested with many combinations of subcommand and subcommand flags.
Updates tryselect unit tests to use Python 3.
Differential Revision: https://phabricator.services.mozilla.com/D73398
This was an issue around "six" and "click", this thread sheds some light:
https://github.com/pallets/click/issues/564.
This issue was reproducible by having a local python 2 flask app with Flask==1.0.2, Werkzeug==0.14.1 and Click==7.0. Upon self-shutdown, the TypeError was raised.
Upgrading Click to 7.1.2 resolves this issue both in that separate "reproduce app" and here in-tree.
Differential Revision: https://phabricator.services.mozilla.com/D73351
This ensures we don't run every build with every push via ./mach try auto. It
introduces a new 'optimization-overrides' try_config that can be used to
replace optimizations. For now, there is no user interface to pass this in via
the 'mach try' command line.
Differential Revision: https://phabricator.services.mozilla.com/D68207
Changes:
Applies the `filter_tasks_by_blacklist` method to try syntax pushes as well.
- moved `TARGET_TASK_BLACKLIST`and `filter_tasks_by_blacklist` method to live in `taskcluster/taskgraph/target_tasks.py`.
- removed existing filters against `ccov, windows10-aarch64` and `android-hw` filters against try syntax pushes.
- update imports for `fuzzy` and `chooser` selectors to refer to the new location of `filter_tasks_by_blacklist` method.
The reason for moving the logic (again) from `tools/tryselect` to `taskcluster/` is due to the placement of `try_option_syntax` and `target_tasks` files and both of those files handle the processing of `mach try syntax` pushes.
Differential Revision: https://phabricator.services.mozilla.com/D71698
Changes:
- implement filtering of taskgraph items against a list of known task filters in `tasks.py`.
- apply these filters to both `fuzzy` and `chooser` selectors, unless `--full` is specified.
Differential Revision: https://phabricator.services.mozilla.com/D70097
--HG--
extra : moz-landing-system : lando
Changes:
- implement filtering of taskgraph items against a list of known task filters in `tasks.py`.
- apply these filters to both `fuzzy` and `chooser` selectors, unless `--full` is specified.
Differential Revision: https://phabricator.services.mozilla.com/D70097
--HG--
extra : moz-landing-system : lando
This prevents a race condition where the watchman hook can potentially
invalidate the cache in-between the call to 'isfile' and 'getmtime'.
Depends on D69189
Differential Revision: https://phabricator.services.mozilla.com/D69190
--HG--
extra : moz-landing-system : lando
The bugbug scheduler currently chooses which manifests are important, and we
then run *every* task that contains those manifests. This is likely overkill
and we can reduce the number of configurations we run these manifests on.
Differential Revision: https://phabricator.services.mozilla.com/D68466
--HG--
extra : moz-landing-system : lando
This ensures we fail with invalid module paths early. Otherwise users wouldn't
find out until the decision task fails.
Differential Revision: https://phabricator.services.mozilla.com/D68464
--HG--
extra : moz-landing-system : lando
These routes are applied to all tasks except the decision task. There
is currently no validity checking in the frontend, so if the provided
routes aren't valid the decision task will fail.
Differential Revision: https://phabricator.services.mozilla.com/D67828
--HG--
extra : moz-landing-system : lando
This adds an optional routes key to the task_task_config schema,
which is a list of strings. Anything in this key is added to the list
of routes for tasks scheduled by the decision task.
Differential Revision: https://phabricator.services.mozilla.com/D67827
--HG--
extra : moz-landing-system : lando
The bugbug scheduler currently chooses which manifests are important, and we
then run *every* task that contains those manifests. This is likely overkill
and we can reduce the number of configurations we run these manifests on.
Differential Revision: https://phabricator.services.mozilla.com/D68466
--HG--
extra : moz-landing-system : lando
This ensures we fail with invalid module paths early. Otherwise users wouldn't
find out until the decision task fails.
Differential Revision: https://phabricator.services.mozilla.com/D68464
--HG--
extra : moz-landing-system : lando
I'm unsure of the root cause as the file should exist if generate_tasks has worked, but this should avoid errors in the meantime while we investigate.
Differential Revision: https://phabricator.services.mozilla.com/D67833
--HG--
extra : moz-landing-system : lando
Neither of these selectors involve the user choosing tasks, so showing
estimates or saving them in the 'mach try again' history doesn't make much
sense.
Differential Revision: https://phabricator.services.mozilla.com/D66628
--HG--
extra : moz-landing-system : lando
This allows us to change the default optimization strategy used in try pushes.
While probably not super useful to developers, it can help us easily test
changes to new and experimental optimizations on try.
This also changes the default to the 'bugbug_push_schedules' strategy, since
SETA is more or less random and shouldn't be used by 'mach try auto'. In the
future, we'll switch this back to simply using the default optimization as the
default will ideally be the best one that we have.
Differential Revision: https://phabricator.services.mozilla.com/D65746
--HG--
extra : moz-landing-system : lando
The 'auto' in 'mach try auto' stands for two things:
1. It automatically picks tasks for you.
2. It runs the same scheduling algorithms as autoland.
It accomplishes this by creating a new target_tasks method that spoofs the
'project' parameter to autoland.
Differential Revision: https://phabricator.services.mozilla.com/D60184
--HG--
extra : moz-landing-system : lando
Add filename support for plain display
Update fzf version in tests
Update tooltool url for fzf
Differential Revision: https://phabricator.services.mozilla.com/D66571
--HG--
extra : moz-landing-system : lando
Remove use of requests module in preview pane
Reformat task duration data to avoid reprocessing in preview pane
Avoid loading task durations json more than once.
Increase required fzf version, use temporary file instead of arglist
Differential Revision: https://phabricator.services.mozilla.com/D65094
--HG--
extra : moz-landing-system : lando
Changes:
Remove `ubuntu-bionic` flag that was used during development to enable use of ubuntu1804-test docker image.
Remove unnecessary conditional and in the process rewrite how `runtests.py` checks the environment for `pactl` prior to running mochitests.
Differential Revision: https://phabricator.services.mozilla.com/D65389
--HG--
extra : moz-landing-system : lando
In |mach try fuzzy| there's a TARGET_TASK_FILTERS variable that we use to make
it more difficult to run certain kinds of expensive / resource constrained
tasks. The problem is that the only way to run these is to use '--full' at
which point there's no way to distinguish which tasks are "valid" (i.e run on
mozilla-central) and which tasks don't.
This adds a flag that will ensure the default set truly matches what we run
on central.
Differential Revision: https://phabricator.services.mozilla.com/D64648
--HG--
extra : moz-landing-system : lando
Ricky Stewart
Andrew Halberstadt
Kartikaya Gupta
Julian Descottes
James Graham
Andrew Halberstadt
Mihai Alexandru Michis
Sonia Singla
Sylvestre Ledru
championshuttler
Tom Prince
Ben Hearsum
Narcis Beleuzu
chocos
Tom Ritter
Mitchell Hentges
Nicolas Silva
Geoff Brown
Marco Castelluccio
Tarek Ziadé
Matthew Gaudet
Bob Clary
Joel Maher
Bogdan Tara
Edwin Takahashi
Csoregi Natalia
Simon Fraser
Mark Banner
Andrew McCreight
Ciure Andrei