We currently vary the cache name for run-task tasks whenever run-task
changes. This allows us to not worry about backwards or forwards
compatibility of caches in run-task tasks.
This strategy doesn't work for out-of-tree Docker images because
the content of run-task cannot be determined at Taskgraph time:
the content of run-task was determined when that Docker image was
built and there is no way to get that content efficiently during
Taskgraph.
So, for out-of-tree Docker images we now vary the cache name by
the Docker image value, which includes its name and a tag or
hash. This means that out-of-tree run-task tasks will get separate
caches for each distinct Docker image.
This isn't ideal. Ideally we would share caches if run-task doesn't
vary between Docker images. But without any way of proving that
at Taskgraph time, we take the safe road and force cache separation.
MozReview-Commit-ID: FMiQBqfvjqW
--HG--
extra : rebase_source : b2763625a3a69e0cf11b6d648a6fcca379234f02
The image_builder Docker image doesn't set a "command" in its task
definition. The image instead relies on a RUN in its Dockerfile to
control the started command. This command is a shell script which
eventually runs run-task.
This all means that image_builder tasks are executing run-task but
the cache sanitization implemented in bug 1391476 isn't getting
applied to those tasks. This means run-task could barf due to
constraint violations due to improperly configured caches.
The fix for this is to teach the generic task transform that
image_builder tasks use run-task. The effect of this is that
some environment variables get set and the cache name changes
depending on the contents of run-task.
MozReview-Commit-ID: IFqsDxD0eDh
--HG--
extra : rebase_source : 280983eae7d6a44dfd70f0da8ce325e90e9555c4
This creates a new "job-from" field that contains the relative filename the job was defined
in. The filename is relative to 'config.path'. If the task came from the 'jobs' key defined
in kind.yml, this field will be set to 'kind.yml'.
MozReview-Commit-ID: 9e1tEb6XuZT
--HG--
rename : taskcluster/ci/source-test/python-tests.yml => taskcluster/ci/source-test/python.yml
extra : rebase_source : 89010f00a96f88a4eecc8b37028f26b483b820ec
The Sphinx documentation only needs access to a relatively small number
of files in the repo in order to be generated. It is a good candidate
for using sparse profiles.
This commit defines and uses a "sphinx-docs" sparse profile containing
only the files relevant to Sphinx documentation generation.
There are some quirks with the profile:
* All moz.build files are included. This bloats the profile
by >1000 files. Worse, it realizes directories that have no business
being realized. This clutters the checkout and makes it harder to
find things. There is a moz.build reader that knows how to retrive
file data from Mercurial. We could use that. This feels like follow-up
fodder.
* All mach_commands.py files are included. `mach help` says you can do
things that you aren't able to do in the sparse checkout. There isn't
a good way to add all *.py files while excluding mach_commands.py
files. We /could/ do it with regular expressions. But those are slow.
Let's leave it as is for now and come up with a better solution later.
MozReview-Commit-ID: 7yiqGGE1nAh
--HG--
extra : rebase_source : c148040ea3618e8bfdd369b6f48fc60c6d179285
extra : source : b76e2f6204b20de137f2566dff8121ff3abe5760
Clean up and standardize Treeherder symbols for Talos and AWSY tasks:
* Stylo disabled groups include `sd`
* Stylo sequential groups include `ss`
MozReview-Commit-ID: 7cl6e0XvXNO
--HG--
extra : rebase_source : a7b4e7c78af673e204d7ff7479ce5fc87614248c
Convert all jobs that were exercising Stylo enabled to Stylo disabled instead.
Stylo enabled is now handled by the default jobs.
In Perfherder, Stylo enabled jobs will be untagged and take over the existing
Gecko series. Stylo disabled jobs will have a new `stylo-disabled` tag and
create a new series.
MozReview-Commit-ID: BMXBRg3A95j
--HG--
extra : rebase_source : 0f3a6a1f6807dbb90c00d060a3a7f220c2141c64
Make Linux Stylo Talos jobs explicit in the task list, like they are for macOS
and Windows. This way we avoid have different approaches per platform.
(This is still the Stylo enabled form. The next patch coverts to Stylo
disabled for all platforms.)
MozReview-Commit-ID: K4CcWvR9tlJ
--HG--
extra : rebase_source : 3727ddb09114e56493b5ee42df1b45a830a3a6dc
We no longer need to run unit / integration tests on Stylo sequential, but we'd
like to retain it for memory and performance checks.
In addition, we no longer need to explicitly `--enable-stylo` since it is the
default. This also drops the `stylo` tag in Perfherder.
MozReview-Commit-ID: 3OI8Y7Q5PW
--HG--
extra : rebase_source : d80103b3c7991a833dc4603d1df4a6c4b62fd6c9
The signing pool is a limited resource. We already have logic to avoid re-signing
widevine if the .sig files are there, and we're taking advantage of that on linux
and windows. OSX was the lone desktop os still pointing at the unsigned bits for
l10n.
We're pointing at repackage instead of signing because the l10n logic expects a dmg.
MozReview-Commit-ID: FtWa4o5s7aJ
--HG--
extra : rebase_source : f704a97293a231d26a4806543f53fc7e83eb9ca9
Our current sccache build links in openssl's libraries dynamically. The
sonames of the dynamic libraries linked in are specific to the
CentOS/Fedora-ish systems that we build on; attempting to run the
generated sccache binaries on different systems (e.g. Debian-ish) will
result in failure. All of our current automation images are
CentOS-based, but for various reasons, Debian-based images may be used
in the future, and it would be great to have an sccache binary to run on
such systems as well. (It might also be interesting to distribute the
sccache binary we use to local developers as well, but that's a bit
further off.)
Therefore, this patch alters the sccache build on Linux to use static
linking for openssl. We cannot use the system openssl we build on
because the system openssl links to libkrb5, and the distribution we use
for the system images does not provide static libraries of libkrb5.
Building openssl ourself enables us to eliminate the libkrb5 dependency.
An sccache binary from builds with this patch depends on the following
libraries:
froydnj@hawkeye:~$ ldd sccache2/sccache
linux-vdso.so.1 => (0x00007ffe02b39000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff0e7403000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff0e71fb000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff0e6fdd000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff0e6dc6000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff0e69fc000)
/lib64/ld-linux-x86-64.so.2 (0x0000557c8540b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff0e66f2000)
which are standard on any Linux distribution.
This includes adding TASKCLUSTER_VOLUMES to docker image builds directly. The
env variable is not added as part of the task transform because `run-task` is
not in payload.command. In fact, build-image.sh calls run-task after doing
some other housekeeping.
Ideally image builds would be turned into jobs and all of this would occur
automatically, but that turns out to be quite a bit too complex for this
incidental fix -- perhaps best solved in another bug.
MozReview-Commit-ID: FYHvafJras7
--HG--
extra : rebase_source : 4e3b9ae9900727e7932c13ced34b3f8596d755d9
Gregory Szorc
Joel Maher
Dustin J. Mitchell
J. Ryan Stinnett
Andrew Halberstadt
Sebastian Hengst
Nick Fitzgerald
Aki Sasaki
Nathan Froyd
Wes Kocher
Phil Ringnalda
Geoff Brown
Johan Lorenzo
Kartikaya Gupta
Brian Stack