These typically run for 25-50 minutes. So, if they are taking longer than that,
there is likely a problem.
Differential Revision: https://phabricator.services.mozilla.com/D2252
--HG--
extra : moz-landing-system : lando
Currently 'fetch' artifacts are all extracted in the same directory, this could
make the extdir messy, or in the worst case, cause file name collisions.
Some artifacts are ok to extract into the same directory as they're already
bundled within the archive. But other artifacts are not. This patch keeps the
default behaviour (extracting everything into the same directory), but allows
task authors to specify per-artifact directories to extract into.
The syntax is:
path[>dest]@<task>
The 'dest' value will be a subdirectory of the MOZ_FETCHES_DIR environment
variable.
Depends on D2102.
Differential Revision: https://phabricator.services.mozilla.com/D2166
--HG--
extra : moz-landing-system : lando
This refactoring will make it easier to set 'run-on-projects' keys across all talos tasks
in one go (rather than needing to modify each task individually).
The job-defaults are recursively (and naively) merged into the task definitions, so each
task that changes 'run-on-projects' also needs to use the 'by-test-platform' key.
Differential Revision: https://phabricator.services.mozilla.com/D2136
--HG--
extra : moz-landing-system : lando
Fetches no longer need to be artifacts exposed via a 'fetch' task, they can
also be artifacts from a task's dependencies. The new format is:
fetches:
fetch:
- fetch-artifact-1.zip
- fetch-artifact-2.zip
build:
- build-artifact-1.zip
...
Specifying 'build' artifacts to fetch will error out if the task doesn't have
any build dependencies.
The 'fetch' key works the same as before, but it is now a special case. Unlike
'build' (or other dependencies), adding a fetch task's artifact here will
implicitly make our task depend on the corresponding fetch task. It will not
be an error.
Depends on D2028.
Differential Revision: https://phabricator.services.mozilla.com/D2102
--HG--
extra : moz-landing-system : lando
Fetches no longer need to be artifacts exposed via a 'fetch' task, they can
also be artifacts from a task's dependencies. The new format is:
fetches:
fetch:
- fetch-artifact-1.zip
- fetch-artifact-2.zip
build:
- build-artifact-1.zip
...
Specifying 'build' artifacts to fetch will error out if the task doesn't have
any build dependencies.
The 'fetch' key works the same as before, but it is now a special case. Unlike
'build' (or other dependencies), adding a fetch task's artifact here will
implicitly make our task depend on the corresponding fetch task. It will not
be an error.
Depends on D2028.
Differential Revision: https://phabricator.services.mozilla.com/D2102
--HG--
extra : moz-landing-system : lando
The 'use_fetches' transform is currently only being used by toolchain tasks,
but we'd like to expand this to more kinds (like 'test' and 'source_test').
The problem is that 'use_fetches' doesn't have a schema, and assumes things
about the kinds of keys that will be set in the job. For example, it assumes
that job['worker']['env'] is going to be forwarded up to the jobdesc properly.
By moving this transform into the set applied to all 'job' tasks, we:
A) Have a task schema we can reliably depend on
B) Can automatically use it from any 'job' task without kind specific
modifications
Since the toolchain tasks apply the 'job' transforms (almost) right after
the 'use_fetches' transform, this change just works.
Differential Revision: https://phabricator.services.mozilla.com/D2028
--HG--
extra : moz-landing-system : lando
This patch makes the QR test platforms tier-1 by default, and removes
the ad-hoc bits that were making individual QR jobs tier-1 before.
However, it also explicitly downgrades some QR jobs to tier-2 or tier-3;
comments in the yml files indicate why.
MozReview-Commit-ID: 1UfPuhcMvIW
--HG--
extra : rebase_source : a2347f6a5929246aaba7656b59c0b8f7aa4ca081
This is needed to not have a circular kind dependency when we actually spell out all dependencies (in a following patch)
Differential Revision: https://phabricator.services.mozilla.com/D1695
--HG--
rename : taskcluster/ci/repackage-signing/kind.yml => taskcluster/ci/repackage-signing-l10n/kind.yml
extra : rebase_source : c2998ba23f213090d27495eb44c3bde3a1628dff
Test verification backfill specifies --gpu-required for certain types
of tests (depending on test path and/or suite). web-platform-tests do
not recognize --gpu-required. This patch updates the backfill logic
to avoid --gpu-required for wpt.
The actionPerm is for access control, so it must limit access to a specific
callback function, not a name (which can apply to mulitple functions).
To make things nicer, we allow functions to specify their cb_name and default
it to the action name. The decorated function names are not used.
MozReview-Commit-ID: 2oiuXrrw7DE
--HG--
extra : rebase_source : 07b27db25e9c8e3226dc996d3fcef401ca498739
Everything but release-promotion (to be handled in another bug) is generic.
For the moment, these all run with the default repo scopes; once this is
landed, I can start adjusting that and granting the necessary scopes only to
these actions.
MozReview-Commit-ID: IB8OEsfeBpj
--HG--
extra : rebase_source : 6ef1697cf255b579097ef8b85be8f9f62718f548
This carefully maintains tasks as an array by putting the conditional inside of
that array. Note that `[{$if: 'false', then: 1}]` returns `[]` in JSON-e --
the missing `else` branch is treated as a missing array element.
MozReview-Commit-ID: 9ARIxW3gfWo
--HG--
extra : rebase_source : 304ce14ccc9abc9f4f48f3179adb981b5fe55a0e
This changes
File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/actions/cancel_all.py", line 30, in list_group
for task in [t['status'] for t in response['tasks']]:
KeyError: u'tasks'
Into a more understandable error (404, in this case).
MozReview-Commit-ID: 5XnFyxIdRfo
--HG--
extra : rebase_source : 797a3117d3246c962f30980c1658fde3bd366135
This will start to cause an error when a newer shellcheck is available in CI
Differential Revision: https://phabricator.services.mozilla.com/D1940
--HG--
extra : moz-landing-system : lando
The resulting action task isn't useful to the user, so instead we send an email
containing a link to the interaction console.
MozReview-Commit-ID: 5uHnQo9WTF6
--HG--
extra : rebase_source : 1213afa7c53a0bcc4a07c4c2970c7bf21ab3b7f1
This also updates actions to "see through" the conditional. Soon we won't be
using kind=task, so this hack will be less important.
MozReview-Commit-ID: Aa6g9ZqoPMa
--HG--
extra : rebase_source : 7434f2047c48ff0d1fa6de9e3419fb4e0bf0bb72
The resulting action task isn't useful to the user, so instead we send an email
containing a link to the interaction console.
MozReview-Commit-ID: 5uHnQo9WTF6
--HG--
extra : rebase_source : ec52a333582a2778c2cec12d612d681e1a9b1976
We're well overdue for an upgrade of the rust compiler requirements.
Now that we're building with 1.28 (albeit a beta, due to be bumped when
it's released), we can bump the requirement away from 1.24 which is now
old. 1.27 is too new, though, so settle for the older 1.26.
--HG--
extra : rebase_source : a17aa496bf3d4af4d1349d69a637c686c6817d0f
Also include webgl2-deqp, which we would like to run eventually, but not yet.
MozReview-Commit-ID: CY4hYCI95ws
--HG--
extra : rebase_source : 9973df0f905bb65d2e8b8c66a6a57e8869e527c1
Also include webgl2-deqp, which we would like to run eventually, but not yet.
MozReview-Commit-ID: FDWdu1J0end
--HG--
extra : rebase_source : a47d88cb2c5eb82e4dfaa9e58d76acbf0736d35d
This will make sure that when running |mach python-test --python 3| locally,
we only run the tests that also run in CI with python 3 (and therefore pass
presumably).
MozReview-Commit-ID: 3OBr9yLSlSq
--HG--
extra : rebase_source : 456340d0ecdddf1078f2b5b4ebb1eddf3813b26a
Assorted fixes from trawling the sphinx logs - malformed formatting, broken references, leftovers from renaming action-task to action-callback and removing
yaml-templates, docstring fixes to make sphinx happier, and typos.
MozReview-Commit-ID: 6jUOljdLoE2
--HG--
extra : rebase_source : f2a9dbcde5180f760a80f47691a70eba76e58bad
GCC will pick up whatever `as` is first in PATH when trying to assemble
files. It expects this `as` to have at least as many features as the
`as` detected at configure time when GCC was originally built. We
should ensure that GCC is always picking up an appropriate `as` by
adding its base directory to the search path; otherwise, we get peculiar
assembler errors.
Various fixes for the docs to make docs better and sphinx happier
* remove unused targets which were duplicated in balrog, partials
* fix up malformed targets and links
* convert docstrings to comments so sphinx ignores example response in utils/partials.py
* section underlines should match titles
MozReview-Commit-ID: GSYqsocBC4I
--HG--
extra : source : e464e9bbb42fb85170f1ce35a01f25811d753871
We're well overdue for an upgrade of the rust compiler requirements.
Now that we're building with 1.28 (albeit a beta, due to be bumped when
it's released), we can bump the requirement away from 1.24 which is now
old. 1.27 is too new, though, so settle for the older 1.26.
--HG--
extra : rebase_source : c788ef4f7da9949b81df2f0577af6f6039ea63d8
We shouldn't run this on central, as it falls back to the dev configs, and fails.
It should be fine on beta/release/esr60. I had to move this version of the check to its own
kind to avoid the dependency tree bringing in the entire build process. Perhaps we can
refactor later to avoid duplication
Differential Revision: https://phabricator.services.mozilla.com/D1765
--HG--
rename : taskcluster/ci/release-bouncer-check/kind.yml => taskcluster/ci/bouncer-check/kind.yml
yaml.load() can evaluate arbitrary Python code via syntax such as
`!!python/object/apply:os.system`. Seriously.
Let's switch taskgraph to yaml.safe_load(), which is reasonable
about limiting magic.
Differential Revision: https://phabricator.services.mozilla.com/D1736
The max-run-time for tasks appears to have been mostly cargo
culted. In particular, a value of 36000 (10 hours) is quite absurd:
no single task takes that long to execute.
This commit reduces the max-run-time of several tasks to
more reasonable values. The goal here is to prevent
excessively long-running tasks. There is definitely room to
further tweak values to further mitigate long-running tasks. But
let's bite off the biggest chunk first, since that doesn't
require much mental effort.
There is a possibility I overshot on some of these tasks. If we
get timeouts, we can always increase the timeout again.
Differential Revision: https://phabricator.services.mozilla.com/D1716
The by-project and by-build-platform are collapsed by resolve_keyed_by(), and in turn by keymatch() in
utils/attributes.py. That needs a regexp to look for matches (if a simple match isn't possible). Rev
39eed777c0c4 missed that when it landed. The trailing hypens are to distringuish between the two linux
platforms.
MozReview-Commit-ID: LiAJpfV71Ws
--HG--
extra : rebase_source : 6952e4b6be4b05cc7917efba8a346211d1e5c7e0
Instead of clang 4, which they were the last to use, so remove the
clang 4 toolchain.
--HG--
extra : rebase_source : d03a083e9217aeb6c1d2c91decb978426f0e8d1a
For those toolchains built with clang, use clang 6.
The only jobs remaining to use the clang 3.9 toolchain are the
base-toolchains build jobs.
--HG--
extra : rebase_source : 1fe92e9e2730c11ce5ce87dddd6496228c3d270c
Many builds have been using clang 6 explicitly because they needed
features or fixes from it that weren't available in earlier versions.
Now that other builds have kept up, it's probably desirable to keep
everything in sync.
--HG--
extra : rebase_source : deb9daaf792d05518e17b1c48589a3b33035a7ab
The linux64-clang toolchain alias is currently clang 5. Switch it to
clang 6, but keep the spidermonkey tsan builds on clang 5 because of
bug 1467673.
The LLVM headers that come with clang 6 contain a DEBUG define that
conflicts with our DEBUG define and breaks the clang-plugin build,
so force unset ours.
--HG--
extra : rebase_source : aae88f1166108f003b06c022f14d5f4c61fc1ed9
Chains a release-eme-free-repack-beetmover-checksums kind after release-eme-free-repack-beetmover, to move
the target.checksums generated by the latter into the beetmover-checksums/ in candidates directory. Those
are then consumed by release-generate-checksum kind.
A lot of details like scopes, worker & provisioner, attributes, as well as data like repack_id and
partner_path, are inherited directly from the parent beetmover task. Mainly to avoid recalculating them.
In contrast to nightly builds, GPG signing of target.checksums has not been implemented. I don't believe
that adds any value in our current system because the sigs are not verified.
MozReview-Commit-ID: 38iz3J2PAXh
--HG--
extra : rebase_source : 8f2bebe747d97437780f1bc0d9e2f42159aee8a6
Any run-task task could exit with EXIT_PURGE_CACHES and expect the
task to be retried. Before this commit, we only registered this exit
code for tasks "using: run-task." That excluded mozharness and
a handful of other "using" flavors.
With this commit, we register the exit codes for all run-task tasks
by sniffing for the presence of run-task in the low-level code that
emits the task definition. This should prevent run-tasks tasks
from falling through the cracks and not having their exit codes
result in meaningful behavior.
Differential Revision: https://phabricator.services.mozilla.com/D1715
This is just enough to make these repos discoverable and suggest their usage.
Anything more would duplicate documentation in those repositories.
MozReview-Commit-ID: 1WNEsoQBB9U
--HG--
extra : rebase_source : 19c25ee4b05fe72d2fed37ca68f75db3918d1c2f
90dca0906337 accidentally broke `mach artifact toolchain --from-build`
because that code is attempting to load toolchain tasks in isolation.
The new "use_fetches" transform added to toolchain tasks requires
that "fetch" tasks are already processed and their references are
available to toolchain tasks.
This commit adds a mechanism to effectively disable the "use_fetches"
transform when called by `mach artifact toolchain`. It is a hack. I
suspect future planned work around artifacts/fetches will necessitate
additional changes to the `mach artifact toolchain` code. But this
can be deferred to a later day: this commit unbusts `mach artifact
toolchain` and isn't super hacky, so it seems more reasonable than
backing out fetch tasks completely.
Differential Revision: https://phabricator.services.mozilla.com/D1588
Andi-Bogdan Postelnicu
Kartikaya Gupta
Roland Mutter Michael
Cosmin Sabou
Tom Prince
Nathan Froyd
Christian Holler
Tiberius Oros
Bogdan Tara
Marco Castelluccio
James Graham
Joel Maher
Andrew Halberstadt
Margareta Eliza Balazs
Tudor-Gabriel Vîjială
Csoregi Natalia
Noemi Erli
Geoff Brown
Ryan VanderMeulen
Andreea Pavel
Justin Wood
David Major
Chris Manchester
Dustin J. Mitchell
Nick Thomas
Henrik Skupin
Simon Fraser
shindli
Chris AtLee
Brindusan Cristian
arthur.iakab
Benjamin Bouvier
Ciure Andrei
Lisa Guo
Mike Hommey
Jeff Gilbert
Gurzau Raul
Aki Sasaki
Rob Wood
Mihai Tabara
Dorel Luca
rforbes
Dave Hunt
Tom Ritter
Gregory Szorc
Bob Clary
Coroiu Cristina
Sebastian Hengst
Brian Stack
Eric Rahm
Bhavesh
Narcis Beleuzu
Ben Hearsum
Steve Fink