`flake8`, `black`, and `pylint` are all installed in `virtualenv`s when running `mach lint`, so installing these packages globally on image setup isn't necessary.
Differential Revision: https://phabricator.services.mozilla.com/D87904
We want to be able to retroactively tell whether a push was a backstop or not.
This patch stores whether or not a push was a "backstop" directly in the
parameters. The optimization strategy now simply returns 'not
params["backstop"]'.
For simplicity, I'm not counting the 'optimized-backstop' as a backstop. It's
unclear if we'll want to be able to detect these types of the pushes in the
future or not, but we can cross that bridge when we get there.
Differential Revision: https://phabricator.services.mozilla.com/D88151
In the past, the 'backstop' optimization was applied to tasks by default across
all projects, even though it only really made sense on autoland. To choose what
would happen on non-autoland branches, we invented this 'remove_on_projects'
concept.
These days, we only apply the backstop optimization in the first place for
autoland. So 'remove_on_projects' is no longer necessary.
Depends on D88149
Differential Revision: https://phabricator.services.mozilla.com/D88150
This patch cleans up some of the backstop strategy names. Specifically:
1. Rename 'full-backstop' -> 'backstop'. The old 'backstop' algorithm was
unused anyway, so there is no conflict. It is also just defined directly in
the decorator rather than using 'Alias'.
So now rather than 'full-backstop' and 'optimized-backstop', it's just
'backstop' and 'optimized-backstop'.
2. Remove 'backstop-X-hours-Y-minutes' strategies, and replace them with
the corresponding 'push-interval-X' strategy.
This means we lose the time component in the 'optimized-backstop'. But it isn't
a problem, because we shouldn't be using a time component there at all anyway
(we should just use it with the 'backstop').
Differential Revision: https://phabricator.services.mozilla.com/D88149
Using nested Any generates deeply nested non-trivial schemas because we
also end up nesting optionally_keyed_by's, and the addition of one field
in one optionally_keyed_by in bug 1657769 made the whole schema
compilation step an order of magnitude slower.
Even before bug 1657769, the schema compilation step was slow for tests,
and with this change, it's virtually instantaneous.
Differential Revision: https://phabricator.services.mozilla.com/D88543
Certain tasks are meant to perform follow-up work to another task. For example,
a 'signing' task signs an artifact from a dependency task. An 'upload-symbols'
tasks uploads an artifact from a dependency task.
In general, we only want to run these types of tasks when the dependency task
is running. But in practice, these tasks often cause the dependency to get
pulled in.
We've tried to fix this in the past by making the 'run-on-projects' and
'optimization' keys match their primary dependency. But it's very easy to
mess this up, and some optimizations (like bugbug) take the content of the
task definition into account. So even if both tasks use the same optimization,
they could have different results.
This revision adds a new 'if-dependencies' key that denotes a task that should
only run if one of the dependencies specified in this list is also run. It
will allow us to stop playing whack-a-mole when trying to make sure tasks
aren't being pulled in solely due to these types of dependencies.
This feature is implemented independently of the 'optimization', so tasks that
specify this can still be optimized even if their primary dependency does run.
Differential Revision: https://phabricator.services.mozilla.com/D80508
Edges are defined as a (left, right, name) tuple, which means task left depends on task right using an edge called name. Previously the optimization code never took dependencies into account, so the mock graph generator never bothered to create them. But in the child revision, we start using these values, so the generation should set it up appropriately.
I had to make it conditional due to the way the assertions in the test_get_subgraph test are set up. I figured it was easier to do that than to re-write the assertions.
Depends on D88483
Differential Revision: https://phabricator.services.mozilla.com/D88484
This changes most of our automation builds to clang 11.0.0 rc2.
Not included:
* code coverage builds, per bug 1660341
* mingw builds, which have traditionally been on their own update cadence, and in this case are blocked anyway by bug 1658632
This will leave some unused clang-9 task definitions. I intend to clean them up, but at a later date. For now I want to focus on making sure this update sticks, since patches like this have a tendency to bounce.
Differential Revision: https://phabricator.services.mozilla.com/D88313
This is a fork of the release-partner-repack-beetmover kind and transform. It's modified to cope with having one upstream task with many partner builds, rather than a many beetmover tasks dealing with a single config-platform-locale combination.
Differential Revision: https://phabricator.services.mozilla.com/D87730
A single task is created to do all partner attributions. The partner_attribution transform processes the configuration into an environment variable for the tools/attribution/attribute.py script to use. This is quite verbose so a large number of configurations may cause problems.
Applies the same priority modification to attribution tasks as to partner repacks, to not impede the main part of the graph.
Differential Revision: https://phabricator.services.mozilla.com/D87729
The partner attribution config is stored in the same repository as the repo manifest for partner repacks, but all in attribution_config.yml instead of default.xml. This extends the existing support for using the Github API to read files to retrieve and process the attribution config.
Differential Revision: https://phabricator.services.mozilla.com/D87728
Renames the release_enable_partners parameter to release_enable_partner_repack, and adds release_enable_partner_attribution for attribution. This it to provide support for disabling them independently in main releases, and in respins.
Adds docs for attribution, update docs for repacks.
Hardwire values for the enable params for the respin flavors, other wise read from the input (defaulting to on in promotion, off otherwise).
Fixes up the rebuild-kinds for partner repacks so that they reflect the current set, although the top level may be all that is needed.
Differential Revision: https://phabricator.services.mozilla.com/D87727
This improves the integrity of downloads of upstream artifacts when using fetch-content. If `verify-hash: True` is set on the fetch config, then the chain-of-trust.json of the upstream is used to retieve the expected sha256 of the artifact, and this is checked.
Differential Revision: https://phabricator.services.mozilla.com/D87725
Prior to this patch the task graph would always include a release-partner-repack-<platform> task for all 6 platforms, regardless of what was specified in release_partner_config. This was particularly obvious in the off-cycle respin scenario when a single partner is repacked. By moving and reusing get_repack_ids_by_platform() it's easy to skip unneeded platforms.
Differential Revision: https://phabricator.services.mozilla.com/D87724
This is a fork of the release-partner-repack-beetmover kind and transform. It's modified to cope with having one upstream task with many partner builds, rather than a many beetmover tasks dealing with a single config-platform-locale combination.
Differential Revision: https://phabricator.services.mozilla.com/D87730
A single task is created to do all partner attributions. The partner_attribution transform processes the configuration into an environment variable for the tools/attribution/attribute.py script to use. This is quite verbose so a large number of configurations may cause problems.
Applies the same priority modification to attribution tasks as to partner repacks, to not impede the main part of the graph.
Differential Revision: https://phabricator.services.mozilla.com/D87729
The partner attribution config is stored in the same repository as the repo manifest for partner repacks, but all in attribution_config.yml instead of default.xml. This extends the existing support for using the Github API to read files to retrieve and process the attribution config.
Differential Revision: https://phabricator.services.mozilla.com/D87728
Renames the release_enable_partners parameter to release_enable_partner_repack, and adds release_enable_partner_attribution for attribution. This it to provide support for disabling them independently in main releases, and in respins.
Adds docs for attribution, update docs for repacks.
Hardwire values for the enable params for the respin flavors, other wise read from the input (defaulting to on in promotion, off otherwise).
Fixes up the rebuild-kinds for partner repacks so that they reflect the current set, although the top level may be all that is needed.
Differential Revision: https://phabricator.services.mozilla.com/D87727
This improves the integrity of downloads of upstream artifacts when using fetch-content. If `verify-hash: True` is set on the fetch config, then the chain-of-trust.json of the upstream is used to retieve the expected sha256 of the artifact, and this is checked.
Differential Revision: https://phabricator.services.mozilla.com/D87725
Prior to this patch the task graph would always include a release-partner-repack-<platform> task for all 6 platforms, regardless of what was specified in release_partner_config. This was particularly obvious in the off-cycle respin scenario when a single partner is repacked. By moving and reusing get_repack_ids_by_platform() it's easy to skip unneeded platforms.
Differential Revision: https://phabricator.services.mozilla.com/D87724
This changes most of our automation builds to clang 11.0.0 rc2.
Not included:
* code coverage builds, per bug 1660341
* mingw builds, which have traditionally been on their own update cadence, and in this case are blocked anyway by bug 1658632
This will leave some unused clang-9 task definitions. I intend to clean them up, but at a later date. For now I want to focus on making sure this update sticks, since patches like this have a tendency to bounce.
Differential Revision: https://phabricator.services.mozilla.com/D88313
This patch fixes an issue in the custom-v8 toolchain where there is tooling missing that is needed to build v8 (the libtinfo5 package).
Differential Revision: https://phabricator.services.mozilla.com/D88310
From mwoodrow:
bcv is 'basic compositor video', which explicitly disables acceleration using a
pref and then runs some video tests. For normal -qr platforms this isn't
useful, since the pref would disable WebRender, and it'd be an identical
configuation to the non-qr variant.
When software webrender is enabled, disabling acceleration won't disable WR,
and test remains useful (testing video performance of the software webrender
configuration).
The baseline I want to compare against here is the non-qr bcv, to see if
webrender-software is worse than layers-software.
Differential Revision: https://phabricator.services.mozilla.com/D88192
It's often useful to nest 'by-*' statements in the CI configs. However, calling
'resolve_keyed_by' will keep drilling down and resolving these statements all
at once. Normally this is fine, but sometimes we may want to use, e.g 'by-foo'
when 'foo' hasn't been resolved yet (i.e, it is being set by a later transform
or something like that).
Currently, this means that we'll hit the 'default' case as 'foo' doesn't exist.
This revision adds the ability to 'defer' evaluation of 'foo' until a later
time.
While running into this scenario might be a code smell, re-organizing
transforms to untangle these kinds of ordering dependencies will be a major
undertaking. Plus, there are legitimate reasons we might want one kind of
ordering with certain tasks, then another ordering for other tasks.
Differential Revision: https://phabricator.services.mozilla.com/D88191
This adds toolchain definitions for clang 11.0.0 rc2, so that developers can get a sneak peek, but nothing in automation uses these tasks yet. We'll make the switch in a later patch.
NB: most of `clang.yml` is rote copy-paste, except for `macosx64-clang-11` which makes a deliberate departure, described in a comment.
Differential Revision: https://phabricator.services.mozilla.com/D88189
Also reflects the move from CraneStation to the WebAssembly account. And we need a small tweak to the build script to accommodate one of the changes that got picked up along the way.
Differential Revision: https://phabricator.services.mozilla.com/D88187
This avoids a set of intermittent issues related to `zstd` decompression failures, which in the absence of these changes break the entire build.
This also requires [updating an environment variable](https://github.com/mozilla/sccache/pull/822), which we do in `client.mk` as well as documentation.
Differential Revision: https://phabricator.services.mozilla.com/D88184
In Bug 1466660, we started deleting the fetches after a task had run, to avoid
interference between tasks. It turns out the only tasks this was for were the
`source-test-jsshell` tasks, which were changed to use an absolute directory in
Bug 1465181. However, since Bug 1568460 we've always used a per-task directory
for fetches, so can remove the work-around of removing fethes.
Differential Revision: https://phabricator.services.mozilla.com/D86670
We define a value `run.workdir` to point to the defaut directory to use for
all sorts of paths, including checkouts, in the job transform. However, that
directory only makes sense for docker-worker tasks, and relative paths should
be used elsewhere. Adjust the paths on non-docker-worker linux tasks to match
macOS paths.
Differential Revision: https://phabricator.services.mozilla.com/D86668
At the upcoming upgrade to clang-11, we'll need to keep code coverage builds on clang-9, so that their file format is understandable our currently LLVM-9-based rustc.
As currently written this change is NFC, but for the future it will opt-out ccov builds from the retargeting of the `linux64-clang` alias.
The macosx.yml change looks larger than it really is, it's mostly pushing defaults down into the tasks so that coverage builds can override them.
Differential Revision: https://phabricator.services.mozilla.com/D87901
The Firefox build requires features of Rust 1.43. However, a given Rust 1.43 nightly might be missing components that exist in the stable release.
By bumping the nightly version in the toolchain to 1.44 (first released on March 18), we ensure that we won't run into an unexpected missing feature.
Depends on D87650
Differential Revision: https://phabricator.services.mozilla.com/D87651
This is needed as future revisions will refactor the optimization process to be
two pass. So we can't be sure if a task is optimized or not until the very end.
Depends on D87771
Differential Revision: https://phabricator.services.mozilla.com/D87772
I got tripped up by the docs for Graph.visit_preorder/visit_postorder which
talks about nodes linking to and from other nodes, but makes no mention of
which direction the linking happens practically (in my mind I thought "linking
to" meant going from task -> dependent).
This blurb would have helped me catch my mistake.
Differential Revision: https://phabricator.services.mozilla.com/D87771
taskgraph-gen.py dies on try.yml currently.
I'm not sure what the underlying issue is, but this gets things working again.
Differential Revision: https://phabricator.services.mozilla.com/D87691
Fixing the font cache issue the 3rd time now by just running "fc-cache" as the worker user should hopefully fix it for real, and no longer be regressed by some docker image changes.
The Puppeteer jobs will keep the 15s startup timeout for Firefox, so that we could detect another fallout again.
Differential Revision: https://phabricator.services.mozilla.com/D87745
In Bug 1466660, we started deleting the fetches after a task had run, to avoid
interference between tasks. It turns out the only tasks this was for were the
`source-test-jsshell` tasks, which were changed to use an absolute directory in
Bug 1465181. However, since Bug 1568460 we've always used a per-task directory
for fetches, so can remove the work-around of removing fethes.
Differential Revision: https://phabricator.services.mozilla.com/D86670
We define a value `run.workdir` to point to the defaut directory to use for
all sorts of paths, including checkouts, in the job transform. However, that
directory only makes sense for docker-worker tasks, and relative paths should
be used elsewhere. Adjust the paths on non-docker-worker linux tasks to match
macOS paths.
Differential Revision: https://phabricator.services.mozilla.com/D86668
In Bug 1466660, we started deleting the fetches after a task had run, to avoid
interference between tasks. It turns out the only tasks this was for were the
`source-test-jsshell` tasks, which were changed to use an absolute directory in
Bug 1465181. However, since Bug 1568460 we've always used a per-task directory
for fetches, so can remove the work-around of removing fethes.
Differential Revision: https://phabricator.services.mozilla.com/D86670
We define a value `run.workdir` to point to the defaut directory to use for
all sorts of paths, including checkouts, in the job transform. However, that
directory only makes sense for docker-worker tasks, and relative paths should
be used elsewhere. Adjust the paths on non-docker-worker linux tasks to match
macOS paths.
Differential Revision: https://phabricator.services.mozilla.com/D86668
This resolves a long-standing issue in development where `mach artifact` (and therefore `mach bootstrap`) would fail unpredictably if you had dirty, but ignored, files in your checkout. Resolving this problem often required unwieldy `hg purge`/`git ignore` incantations that are easy to get wrong.
This patch addresses the problem by doing what we "should" have been doing all along, and consulting the VCS to list tracked files rather than listing EVERY file on disk and applying heuristics to determine whether they should be included in the hash.
Differential Revision: https://phabricator.services.mozilla.com/D86780
Today we don't require that `mach` `CommandProvider`s subclass from any particular parent class and we're very lax about the requirements they must meet. While that's convenient in certain circumstances, it has some unfortunate implications for feature development.
Today the only requirements that we have for `CommandProvider`s are that they have an `__init__()` method that takes either 1 or 2 arguments, the second of which must be called `context` and is populated with the `mach` `CommandContext`. Again, while this flexibility is occasionally convenient, it is limiting. As we add features to `mach`, having a better idea what the shape of our `CommandProvider`s are and how we can instantiate them and use them is increasingly important, and this gives us additional control when having `mach` configure `CommandProvider`s based on data that is only available at the `mach` level. In particular, we plan to leverage this in bugs 985141 and 1654074.
Here we add validation to the `CommandProvider` decorator to ensure all classes inherit from `MachCommandBase`, update all `CommandProvider`s in-tree to inherit from `MachCommandBase`, and update source and test code accordingly.
Follow-up work: we now require (de facto) that the `context` be populated with a `topdir` attribute by the `populate_context_handler` function, since instantiating the `MachCommandBase` requires a `topdir` be provided. This is fine for now in the interest of keeping this patch reasonably sized, but some additional refactoring could make this cleaner.
Differential Revision: https://phabricator.services.mozilla.com/D86255
This adds a new job variant `arm64-cranelift-sim` to the SpiderMonkey
build configurations, and adds a Taskherder CI configuration to run it.
The job uses the aarch64 simulator support built-in to SpiderMonkey, so
it does not need to run on native aarch64 hardware.
A few tests needed to be added to the "slow tests" list as they time out
under the simulator otherwise.
This also fixes an issue with an error message in `build-sm.sh` in which
the overloading of the backtick's meaning (code-quotes in
Markdown-world, and command interpolation in shell scripts) led to an
amusing attempt to execute parts of the error message.
Finally, this fixes an error that seems unrelated to Cranelift or
WebAssembly in a GC jit-test, wherein its way of measuring maximum stack
recursion depth was failing.
Differential Revision: https://phabricator.services.mozilla.com/D86131
I've left the monitor disabled for now, so that we can have a smaller pushes for enabling and disabling it if needed. It should allow more fine grained control.
We may also want to include extracting the monitor tool from a github version instead, and also removing the assumption and it being forked from the parent, so that it's instead given a process ID to treat as the parent it should watch.
Differential Revision: https://phabricator.services.mozilla.com/D84374
In bug 1626058, I changed how the docker image digest was generated:
- I used the same directory structure to generate the digest as was used for generating the context
- I moved context generation to the decision task, and used the hash of that as part of the digest.
Unfortunately, it turns out the file name in the gzip header of the context
.tar.gz differed between when we are creating a context to write out, and when
were just generating the hash.
This adjust the name used in the gzip header to be consistent.
Differential Revision: https://phabricator.services.mozilla.com/D84753
For all fission web-platform tests currently running on mozilla-central as tier 2,
promote to tier 1 and run on all trunk trees.
Differential Revision: https://phabricator.services.mozilla.com/D85512
This patch fixes an issue where the metric settings were not being used because they don't use the test name. It also handles some changes (from a bad copy-paste) that didn't make it into the last live-site patch series.
Differential Revision: https://phabricator.services.mozilla.com/D85609
We need this bump for a couple of reasons:
1. It enables floating the libc crate version, which makes keeping up-to-date
with upstream changes easier.
2. It enables floating the getrandom crate version, which enables updating
getrandom to a version with less buggy behavior around the Linux
`getrandom` syscall on some architectures.
We also take this opportunity to sync up our vendored lucet with the lucet
that we use to build things in taskcluster.
Differential Revision: https://phabricator.services.mozilla.com/D85408
Sets autoland to use the 'bugbug' test manifest loader. This is being enabled
as part of a temporary trial to see the impact it has on sheriffing.
Differential Revision: https://phabricator.services.mozilla.com/D85011
The enhancements in Bug 1653986 to display information about tests derives its
data from these 2 jobs and so it's appropriate to explicitly depend on them.
The current status of these jobs in the tree as far as I can tell is that:
- "source-test-file-metadata-test-info-all": Seems to get automatically
triggered by someone's regularly, so it's always likely to be there for
our searchfox cron jobs, but it's nice to not depend on that.
- "source-test-wpt-metadata-summary" : Seems to get automatically run, but
only on commits that touch meta files change, per
https://searchfox.org/mozilla-central/rev/cf561cece0ca9aeaf0202e68699836d957d0c670/taskcluster/ci/source-test/wpt-metadata.yml#31
and indeed it wasn't there on today's searchfox jobs.
Differential Revision: https://phabricator.services.mozilla.com/D84933
Sets autoland to use the 'bugbug' test manifest loader. This is being enabled
as part of a temporary trial to see the impact it has on sheriffing.
Differential Revision: https://phabricator.services.mozilla.com/D85011
This creates a new set of optimization strategies
(taskgraph.optimize:project.autoland) to use with autoland. Among other things,
it also means there's no need for the 'test-try' optimization as the autoland
strategies are no longer the default behaviour.
Differential Revision: https://phabricator.services.mozilla.com/D79705
Creates a new utility function in 'taskgraph.util.taskcluster' to return task
definitions from a task group. Also refactors the existing
'list_task_group_incomplete_tasks' function to use it.
Differential Revision: https://phabricator.services.mozilla.com/D74407
Windows jobs typically take 23 minutes, so limit to 30 minutes so that cloning hangs terminate earlier than 2 hours.
Mac and Linux jobs are typically 5 minutes long, so bring the limit in to 15 minutes from an hour.
Also refactors so this the max run time is only specified in the kind, rather than some in the kind and some in the transform.
Differential Revision: https://phabricator.services.mozilla.com/D85077
This adds a makecab toolchain for Linux. It's not hooked anywhere
because bug 1654994 will also move the use of makecab to upload-symbols
tasks, so hooking the toolchain up with builds would be a waste of
time.
Differential Revision: https://phabricator.services.mozilla.com/D84789
It could go into its own test suite, but it 1) depends on `mozbuild` code, so the `mozbuild` suite as well as this new suite would be running on any push that touches `mozbuild` code anyway, and 2) this is code that runs during the build, so it's not out of place.
Differential Revision: https://phabricator.services.mozilla.com/D84547
It could go into its own test suite, but it 1) depends on `mozbuild` code, so the `mozbuild` suite as well as this new suite would be running on any push that touches `mozbuild` code anyway, and 2) this is code that runs during the build, so it's not out of place.
Differential Revision: https://phabricator.services.mozilla.com/D84547
The Reference Browser represents our tech stack up to and including the Android-Components layer.
It would be useful to have an applink startup test for this browser to detect improvements and regressions.
Differential Revision: https://phabricator.services.mozilla.com/D81407
This creates a new set of optimization strategies
(taskgraph.optimize:project.autoland) to use with autoland. Among other things,
it also means there's no need for the 'test-try' optimization as the autoland
strategies are no longer the default behaviour.
Differential Revision: https://phabricator.services.mozilla.com/D79705
Creates a new utility function in 'taskgraph.util.taskcluster' to return task
definitions from a task group. Also refactors the existing
'list_task_group_incomplete_tasks' function to use it.
Differential Revision: https://phabricator.services.mozilla.com/D74407
It could go into its own test suite, but it 1) depends on `mozbuild` code, so the `mozbuild` suite as well as this new suite would be running on any push that touches `mozbuild` code anyway, and 2) this is code that runs during the build, so it's not out of place.
Differential Revision: https://phabricator.services.mozilla.com/D84547
I suspect these were an artifact of building the checked-out repository
inside the Firefox source directory, but that is not a problem anymore.
Differential Revision: https://phabricator.services.mozilla.com/D84363
We don't need this for our (current) builds, which are cross-builds, but we
would need this at some future date if we ditched the breakpad `dump_syms`.
Depends on D83528
Differential Revision: https://phabricator.services.mozilla.com/D83529
I suspect these were an artifact of building the checked-out repository
inside the Firefox source directory, but that is not a problem anymore.
Differential Revision: https://phabricator.services.mozilla.com/D83528
When links to specific tasks in treeherder was added to the description in
Bug 1630710, This caused the descriptions for several derived tasks to include
the `{'task-reference': '...'}` wrapper in the derived description. This is
in addition to the already duplicate links to treeherder those tasks had.
To ameliorate this issue, keep track of the original description without the
treeherder link and use that in derived tasks.
Differential Revision: https://phabricator.services.mozilla.com/D84047
This adds the talos-profiling jobs to the relevant test-platforms so that they
can be run for Linux and Windows shippable builds with WebRender enabled.
However they are also disabled from running on m-c by default; they must be
run explicitly using `./mach try fuzzy --full` or using the TreeHerder add jobs
feature.
Differential Revision: https://phabricator.services.mozilla.com/D84011
This block is identical to the run-on-projects applied to all the talos
profiling jobs in the defaults section at the top of the file. Therefore
it is redundant.
Differential Revision: https://phabricator.services.mozilla.com/D84010
Since the system mozdevice version on bitbar is old, and mozharness now requires
mozdevice 4.0.0, we simply have the build job make a tarball of mozdevice 4.0.0
from the gecko source tree and export it as an artifact. The wrench job that
runs on bitbar can unpack the tarball and add it to the python path so it gets
picked up.
Differential Revision: https://phabricator.services.mozilla.com/D84070
This change the strategy aliases that have bugbug, so that they handle the extra
arguments that are needed by bugbug.
Differential Revision: https://phabricator.services.mozilla.com/D83438