yaml.load() can evaluate arbitrary Python code via syntax such as
`!!python/object/apply:os.system`. Seriously.
Let's switch taskgraph to yaml.safe_load(), which is reasonable
about limiting magic.
Differential Revision: https://phabricator.services.mozilla.com/D1736
The max-run-time for tasks appears to have been mostly cargo
culted. In particular, a value of 36000 (10 hours) is quite absurd:
no single task takes that long to execute.
This commit reduces the max-run-time of several tasks to
more reasonable values. The goal here is to prevent
excessively long-running tasks. There is definitely room to
further tweak values to further mitigate long-running tasks. But
let's bite off the biggest chunk first, since that doesn't
require much mental effort.
There is a possibility I overshot on some of these tasks. If we
get timeouts, we can always increase the timeout again.
Differential Revision: https://phabricator.services.mozilla.com/D1716
The by-project and by-build-platform are collapsed by resolve_keyed_by(), and in turn by keymatch() in
utils/attributes.py. That needs a regexp to look for matches (if a simple match isn't possible). Rev
39eed777c0c4 missed that when it landed. The trailing hypens are to distringuish between the two linux
platforms.
MozReview-Commit-ID: LiAJpfV71Ws
--HG--
extra : rebase_source : 6952e4b6be4b05cc7917efba8a346211d1e5c7e0
Instead of clang 4, which they were the last to use, so remove the
clang 4 toolchain.
--HG--
extra : rebase_source : d03a083e9217aeb6c1d2c91decb978426f0e8d1a
For those toolchains built with clang, use clang 6.
The only jobs remaining to use the clang 3.9 toolchain are the
base-toolchains build jobs.
--HG--
extra : rebase_source : 1fe92e9e2730c11ce5ce87dddd6496228c3d270c
Many builds have been using clang 6 explicitly because they needed
features or fixes from it that weren't available in earlier versions.
Now that other builds have kept up, it's probably desirable to keep
everything in sync.
--HG--
extra : rebase_source : deb9daaf792d05518e17b1c48589a3b33035a7ab
The linux64-clang toolchain alias is currently clang 5. Switch it to
clang 6, but keep the spidermonkey tsan builds on clang 5 because of
bug 1467673.
The LLVM headers that come with clang 6 contain a DEBUG define that
conflicts with our DEBUG define and breaks the clang-plugin build,
so force unset ours.
--HG--
extra : rebase_source : aae88f1166108f003b06c022f14d5f4c61fc1ed9
Chains a release-eme-free-repack-beetmover-checksums kind after release-eme-free-repack-beetmover, to move
the target.checksums generated by the latter into the beetmover-checksums/ in candidates directory. Those
are then consumed by release-generate-checksum kind.
A lot of details like scopes, worker & provisioner, attributes, as well as data like repack_id and
partner_path, are inherited directly from the parent beetmover task. Mainly to avoid recalculating them.
In contrast to nightly builds, GPG signing of target.checksums has not been implemented. I don't believe
that adds any value in our current system because the sigs are not verified.
MozReview-Commit-ID: 38iz3J2PAXh
--HG--
extra : rebase_source : 8f2bebe747d97437780f1bc0d9e2f42159aee8a6
Any run-task task could exit with EXIT_PURGE_CACHES and expect the
task to be retried. Before this commit, we only registered this exit
code for tasks "using: run-task." That excluded mozharness and
a handful of other "using" flavors.
With this commit, we register the exit codes for all run-task tasks
by sniffing for the presence of run-task in the low-level code that
emits the task definition. This should prevent run-tasks tasks
from falling through the cracks and not having their exit codes
result in meaningful behavior.
Differential Revision: https://phabricator.services.mozilla.com/D1715
This is just enough to make these repos discoverable and suggest their usage.
Anything more would duplicate documentation in those repositories.
MozReview-Commit-ID: 1WNEsoQBB9U
--HG--
extra : rebase_source : 19c25ee4b05fe72d2fed37ca68f75db3918d1c2f
90dca0906337 accidentally broke `mach artifact toolchain --from-build`
because that code is attempting to load toolchain tasks in isolation.
The new "use_fetches" transform added to toolchain tasks requires
that "fetch" tasks are already processed and their references are
available to toolchain tasks.
This commit adds a mechanism to effectively disable the "use_fetches"
transform when called by `mach artifact toolchain`. It is a hack. I
suspect future planned work around artifacts/fetches will necessitate
additional changes to the `mach artifact toolchain` code. But this
can be deferred to a later day: this commit unbusts `mach artifact
toolchain` and isn't super hacky, so it seems more reasonable than
backing out fetch tasks completely.
Differential Revision: https://phabricator.services.mozilla.com/D1588
1. Updated hgrepo to work with mozilla-beta, mozilla-esr60 and project branches (just in case)
2. Presquashed commits, so we only submit one.
3. Replaced 'which' with 'command -v' to avoid future shellcheck issues.
Differential Revision: https://phabricator.services.mozilla.com/D1582
Currently, many tasks fetch content from the Internets. A problem with
that is fetching from the Internets is unreliable: servers may have
outages or be slow; content may disappear or change out from under us.
The unreliability of 3rd party services poses a risk to Firefox CI.
If services aren't available, we could potentially not run some CI tasks.
In the worst case, we might not be able to release Firefox. That would
be bad. In fact, as I write this, gmplib.org has been unavailable for
~24 hours and Firefox CI is unable to retrieve the GMP source code.
As a result, building GCC toolchains is failing.
A solution to this is to make tasks more hermetic by depending on
fewer network services (which by definition aren't reliable over time
and therefore introduce instability).
This commit attempts to mitigate some external service dependencies
by introducing the *fetch* task kind.
The primary goal of the *fetch* kind is to obtain remote content and
re-expose it as a task artifact. By making external content available
as a cached task artifact, we allow dependent tasks to consume this
content without touching the service originally providing that
content, thus eliminating a run-time dependency and making tasks more
hermetic and reproducible over time.
We introduce a single "fetch-url" "using" flavor to define tasks that
fetch single URLs and then re-expose that URL as an artifact. Powering
this is a new, minimal "fetch" Docker image that contains a
"fetch-content" Python script that does the work for us.
We have added tasks to fetch source archives used to build the GCC
toolchains.
Fetching remote content and re-exposing it as an artifact is not
very useful by itself: the value is in having tasks use those
artifacts.
We introduce a taskgraph transform that allows tasks to define an
array of "fetches." Each entry corresponds to the name of a "fetch"
task kind. When present, the corresponding "fetch" task is added as a
dependency. And the task ID and artifact path from that "fetch" task
is added to the MOZ_FETCHES environment variable of the task depending
on it. Our "fetch-content" script has a "task-artifacts"
sub-command that tasks can execute to perform retrieval of all
artifacts listed in MOZ_FETCHES.
To prove all of this works, the code for fetching dependencies when
building GCC toolchains has been updated to use `fetch-content`. The
now-unused legacy code has been deleted.
This commit improves the reliability and efficiency of GCC toolchain
tasks. Dependencies now all come from task artifacts and should always
be available in the common case. In addition, `fetch-content` downloads
and extracts files concurrently. This makes it faster than the serial
application which we were previously using.
There are some things I don't like about this commit.
First, a new Docker image and Python script for downloading URLs feels
a bit heavyweight. The Docker image is definitely overkill as things
stand. I can eventually justify it because I want to implement support
for fetching and repackaging VCS repositories and for caching Debian
packages. These will require more packages than what I'm comfortable
installing on the base Debian image, therefore justifying a dedicated
image.
The `fetch-content static-url` sub-command could definitely be
implemented as a shell script. But Python is readily available and
is more pleasant to maintain than shell, so I wrote it in Python.
`fetch-content task-artifacts` is more advanced and writing it in
Python is more justified, IMO. FWIW, the script is Python 3 only,
which conveniently gives us access to `concurrent.futures`, which
facilitates concurrent download.
`fetch-content` also duplicates functionality found elsewhere.
generic-worker's task payload supports a "mounts" feature which
facilitates downloading remote content, including from a task
artifact. However, this feature doesn't exist on docker-worker.
So we have to implement downloading inside the task rather than
at the worker level. I concede that if all workers had generic-worker's
"mounts" feature and supported concurrent download, `fetch-content`
wouldn't need to exist.
`fetch-content` also duplicates functionality of
`mach artifact toolchain`. I probably could have used
`mach artifact toolchain` instead of writing
`fetch-content task-artifacts`. However, I didn't want to introduce
the requirement of a VCS checkout. `mach artifact toolchain` has its
origins in providing a feature to the build system. And "fetching
artifacts from tasks" is a more generic feature than that. I think
it should be implemented as a generic feature and not something that is
"toolchain" specific.
I think the best place for a generic "fetch content" feature is in
the worker, where content can be defined in the task payload. But as
explained above, that feature isn't universally available. The next
best place is probably run-task. run-task already performs generic,
very-early task preparation steps, such as performing a VCS checkout.
I would like to fold `fetch-content` into run-task and make it all
driven by environment variables. But run-task is currently Python 2
and achieving concurrency would involve a bit of programming (or
adding package dependencies). I may very well port run-task to Python
3 and then fold fetch-content into it. Or maybe we leave
`fetch-content` as a standalone script.
MozReview-Commit-ID: AGuTcwNcNJR
--HG--
extra : rebase_source : 4918b8c3bac53d63665006802054038bfbca0314
Currently, many tasks fetch content from the Internets. A problem with
that is fetching from the Internets is unreliable: servers may have
outages or be slow; content may disappear or change out from under us.
The unreliability of 3rd party services poses a risk to Firefox CI.
If services aren't available, we could potentially not run some CI tasks.
In the worst case, we might not be able to release Firefox. That would
be bad. In fact, as I write this, gmplib.org has been unavailable for
~24 hours and Firefox CI is unable to retrieve the GMP source code.
As a result, building GCC toolchains is failing.
A solution to this is to make tasks more hermetic by depending on
fewer network services (which by definition aren't reliable over time
and therefore introduce instability).
This commit attempts to mitigate some external service dependencies
by introducing the *fetch* task kind.
The primary goal of the *fetch* kind is to obtain remote content and
re-expose it as a task artifact. By making external content available
as a cached task artifact, we allow dependent tasks to consume this
content without touching the service originally providing that
content, thus eliminating a run-time dependency and making tasks more
hermetic and reproducible over time.
We introduce a single "fetch-url" "using" flavor to define tasks that
fetch single URLs and then re-expose that URL as an artifact. Powering
this is a new, minimal "fetch" Docker image that contains a
"fetch-content" Python script that does the work for us.
We have added tasks to fetch source archives used to build the GCC
toolchains.
Fetching remote content and re-exposing it as an artifact is not
very useful by itself: the value is in having tasks use those
artifacts.
We introduce a taskgraph transform that allows tasks to define an
array of "fetches." Each entry corresponds to the name of a "fetch"
task kind. When present, the corresponding "fetch" task is added as a
dependency. And the task ID and artifact path from that "fetch" task
is added to the MOZ_FETCHES environment variable of the task depending
on it. Our "fetch-content" script has a "task-artifacts"
sub-command that tasks can execute to perform retrieval of all
artifacts listed in MOZ_FETCHES.
To prove all of this works, the code for fetching dependencies when
building GCC toolchains has been updated to use `fetch-content`. The
now-unused legacy code has been deleted.
This commit improves the reliability and efficiency of GCC toolchain
tasks. Dependencies now all come from task artifacts and should always
be available in the common case. In addition, `fetch-content` downloads
and extracts files concurrently. This makes it faster than the serial
application which we were previously using.
There are some things I don't like about this commit.
First, a new Docker image and Python script for downloading URLs feels
a bit heavyweight. The Docker image is definitely overkill as things
stand. I can eventually justify it because I want to implement support
for fetching and repackaging VCS repositories and for caching Debian
packages. These will require more packages than what I'm comfortable
installing on the base Debian image, therefore justifying a dedicated
image.
The `fetch-content static-url` sub-command could definitely be
implemented as a shell script. But Python is readily available and
is more pleasant to maintain than shell, so I wrote it in Python.
`fetch-content task-artifacts` is more advanced and writing it in
Python is more justified, IMO. FWIW, the script is Python 3 only,
which conveniently gives us access to `concurrent.futures`, which
facilitates concurrent download.
`fetch-content` also duplicates functionality found elsewhere.
generic-worker's task payload supports a "mounts" feature which
facilitates downloading remote content, including from a task
artifact. However, this feature doesn't exist on docker-worker.
So we have to implement downloading inside the task rather than
at the worker level. I concede that if all workers had generic-worker's
"mounts" feature and supported concurrent download, `fetch-content`
wouldn't need to exist.
`fetch-content` also duplicates functionality of
`mach artifact toolchain`. I probably could have used
`mach artifact toolchain` instead of writing
`fetch-content task-artifacts`. However, I didn't want to introduce
the requirement of a VCS checkout. `mach artifact toolchain` has its
origins in providing a feature to the build system. And "fetching
artifacts from tasks" is a more generic feature than that. I think
it should be implemented as a generic feature and not something that is
"toolchain" specific.
I think the best place for a generic "fetch content" feature is in
the worker, where content can be defined in the task payload. But as
explained above, that feature isn't universally available. The next
best place is probably run-task. run-task already performs generic,
very-early task preparation steps, such as performing a VCS checkout.
I would like to fold `fetch-content` into run-task and make it all
driven by environment variables. But run-task is currently Python 2
and achieving concurrency would involve a bit of programming (or
adding package dependencies). I may very well port run-task to Python
3 and then fold fetch-content into it. Or maybe we leave
`fetch-content` as a standalone script.
MozReview-Commit-ID: AGuTcwNcNJR
--HG--
extra : source : 0b941cbdca76fb2fbb98dc5bbc1a0237c69954d0
extra : histedit_source : a3e43bdd8a9a58550bef02fec3be832ca304ea93
After this change, we consistently import GPG keys from files in
the GCC build scripts.
MozReview-Commit-ID: BcyvCQoGbMS
--HG--
extra : rebase_source : 657ccce8e242cabdfaff396fd0d6439754a3f364
After this change, we consistently import GPG keys from files in
the GCC build scripts.
MozReview-Commit-ID: BcyvCQoGbMS
--HG--
extra : source : 5fce34a460b51e45ac280a9f0cb8bad896fbcff1
extra : histedit_source : 01621ea8111315c251a9493a11efca72c2ba3c7d
Let's install python-zstandard for both Python 2 and Python 3 in
all our Debian-based images so it is readily available for use.
MozReview-Commit-ID: 1L8zDc5MYXA
--HG--
extra : rebase_source : db718891dd31d4feceff76fbce753b63049e20b1
python-zstandard's 0.9.1 source distribution contains a debian/
directory.
On Squeeze, producing a Debian package is straightforward.
On Wheezy, we need to hack up Build-Depends because Wheezy doesn't
have a package for the Hypothesis fuzzing library. This package is
only used for testing and our package building disables testing,
so we don't even need to further hack up the packaging to disable
tests.
MozReview-Commit-ID: 6raXjdzggCH
--HG--
extra : rebase_source : 672492a40d65df8430eb17ba033bcb1c0890b7df
dh-python isn't available in Wheezy. Let's backport it so we can
build Python packages that use it.
Fortunately for us, the package builds without any modifications.
The only customization we need is to ensure our custom Python
packages are present in order to satisfy Build-Depends.
MozReview-Commit-ID: CqZtwvosA6K
--HG--
extra : rebase_source : 36515905a6c5937ba16f5f4b566b61715b4f26ac
This build target doesn't have LTO enabled on it (yet)
MozReview-Commit-ID: 56tAHMyvH7o
--HG--
extra : rebase_source : 90039cd8e97332e2ef8aad7908b8a04b2869f4a5
They are no longer used. By not listing them, we prevent new tasks
from using them.
MozReview-Commit-ID: FiYhV8WcAsm
--HG--
extra : rebase_source : 18a5be1c050b622f1c57f752b3f97563009cdaf1
For the same reasons that we moved build tasks to this worker type.
MozReview-Commit-ID: LZuzDtHSKL6
--HG--
extra : rebase_source : 6fa455cde001f966024186a26e0589fc9f139293
The gecko-{L}-b-macosx64 worker types are really Linux (macOS builds are
cross-compiled). These worker types are essentially identical to their
gecko-{L}-b-linux counterparts.
I don't see a compelling reason to maintain separate worker types for
these builds other than maybe cost accounting (worker types are tagged
in AWS land and these tags can be more easily broken out for billing
analysis). But I don't think any important systems are relying on
this "feature."
So let's move the macOS build tasks to the gecko-{L}-b-linux workers.
MozReview-Commit-ID: 67bArn6IG9T
--HG--
extra : rebase_source : 4de6bf450e7d0d982a770ca8a92e1ac1982fa228
Task run time for these tests is highly variable across chunks: Some run
in only 30 minutes, while xpcshell-11 sometimes exceeds 90 minutes.
Rather than waste resources on more chunks, I think increasing the max
task run time is a reasonable way of avoiding intermittent task time-outs.
While some builds have a PERFHERDER_EXTRA_OPTIONS environment set on the
taskcluster side, many others have the equivalent set at the mozharness
level. But only the former are actually linted against, which,
unsurprisingly, translates to conflicting values between some of the
mozharness configs.
So we move those configurations to taskcluster, enable the lint on all
the kinds that look like builds (based on them using the build_attrs
transform), and adjust the values to stop conflicting. Notably, for
searchfox and static-analysis-autotest.
--HG--
extra : rebase_source : 097333608e61e1df66e5d8f914e15784f35e58f2
This will get the js-bench tasks to run on physical hardware instead of AWS.
MozReview-Commit-ID: 44XavXAwxxn
--HG--
extra : rebase_source : ae1ba4c7f90b3a8526511a3f3c1dff913a334619
There is a superficial check in the run-task script which requires root. Simply
removing this check allows a native-engine task (which isn't running as root)
to proceed.
MozReview-Commit-ID: 44XavXAwxxn
--HG--
extra : rebase_source : bd1f01ce1c2feb4029838e07314493d449a4f46e
This adds an optional 'workdir' key to all job schemas. It still defaults to
/builds/worker, but can be overriden by individual tasks or schema
implementations.
MozReview-Commit-ID: LY20xfBhbCP
--HG--
extra : rebase_source : 7ac76ebf55d33d30c2aad73484421c6b4002cd33
Extends support of the use-artifacts key to native-engine based tasks.
MozReview-Commit-ID: FJILoyD5XVZ
--HG--
extra : rebase_source : 0cf8bf63f73d0fbb634f6b437bcc9bcce7821900
I've deliberately left as tier-3 the following tests:
- gtest (perma-OOM, likely from ASan malloc-meddling)
- xpcshell (builds need to be signed plus other failures too)
--HG--
extra : rebase_source : 812bf0de11e91c4e952cb5da9163241bd9386246
Summary:
the blocklist and remote-settings changes need to happen on beta,
but not the hsts/hpkp updates, so we have to split out the control of what
runs by project.
Reviewers: jlorenzo
Reviewed By: jlorenzo
Bug #: 1436369
Differential Revision: https://phabricator.services.mozilla.com/D1487
--HG--
extra : rebase_source : 19ccbb67b880ee7bd2dc2a37325dd70de635abad
Joel Maher
Mike Hommey
Tom Ritter
Gregory Szorc
Bob Clary
Rob Wood
Coroiu Cristina
Justin Wood
Csoregi Natalia
Sebastian Hengst
Simon Fraser
Brian Stack
Gurzau Raul
Kartikaya Gupta
Andreea Pavel
Dorel Luca
Nick Thomas
shindli
David Major
Geoff Brown
Ciure Andrei
Eric Rahm
Bhavesh
Brindusan Cristian
Narcis Beleuzu
Chris Manchester
Chris AtLee
Ben Hearsum
Dustin J. Mitchell
Bogdan Tara
Steve Fink
arthur.iakab
Ahilya Sinha
Marco Castelluccio
Cosmin Sabou
Noemi Erli
Tom Prince
Johan Lorenzo
Andrew Halberstadt