`run-task` is taught a --sparse-profile argument to be passed down
to `hg robustcheckout` for the main source checkout. It does what
you expect: performs a sparse checkout using the named profile.
The Taskgraph YAML for run-task is taught a "sparse-profile"
property to define the sparse profile. When defined, --sparse-profile
will be passed down to `run-task` and the cache name will be updated
to reflect the use of sparse checkout.
Our cache checking transform is updated to audit for the use of
--sparse-profile without the corresponding "-sparse" cache name
variation.
The reason we need a distinct cache name for sparse is because
clients that aren't sparse aware will be unable to read checkouts
that are sparse. By forcing sparse and non-sparse into different
cache pools, we avoid compatibility issues.
In the ideal world, we probably support sparse profiles on all the
VCS checkouts that `run-task` supports (e.g. --tools-checkout).
Perfect is the enemy of done. All of this is defined in-tree and
it is easy enough to change atomically.
MozReview-Commit-ID: 79k7Vul0hHO
--HG--
extra : rebase_source : babe9b42e2796c2341bffc6ecfe829f4daff9e0f
Include "error:" so Treeherder log parser picks things up (hopefully).
Also, actually include cache path in string.
MozReview-Commit-ID: 8b8ou7TJYZs
--HG--
extra : rebase_source : 52b43dd18419a6a8cfb096a6bf4262852c7c0ccc
* eliminate heading for test kinds, of which there is now only one
* make the caches document have a single heading in the TOC
* break out mach commands into a separate document, add ./mach taskgraph morphed
* remove docs for YAML templates support (the .yml file wasn't actually
used -- I expect it was a merge leftover); these are still used for actions.yml,
but once that is gone the code should be removed, too.
* break try out into its own document, edit to distinguish "how to run try"
from "how to generate config"
MozReview-Commit-ID: 76ZopWA9TPL
--HG--
extra : rebase_source : 6946d866f9df6eec591b9a05ddedc6467dd69e4b
The UID and GID that a task executes under is dynamic. As a result,
caches need to be aware of the UID and GID that owns files otherwise
subsequent tasks could run into permission denied errors. This is
why `run-task --chown-recursive` exists. By recursively changing
ownership of persisted files, we ensure the current task is able
to read and write all existing files.
When you take a step back, you realize that chowning of cached
files is an expensive workaround. Yes, this results in cache hits.
But the cost is you potentially have to perform hundreds of thousands
of I/O system calls to mass chown. The ideal situation is that
UID/GID is consistent across tasks on any given cache and
potentially expensive permissions setting can be avoided. So, that's
what this commit does.
We add the task's UID and GID to run-task's requirements. When we
first see a cache, we record a UID and GID with it and chown the
empty cache directory to that UID and GID. Subsequent tasks using
this cache *must* use the same UID and GID or else run-task will
fail.
Since run-task now guarantees that all cache consumers use the same
UID and GID, we can avoid a potentially expensive recursive chown.
But there is an exception. In untrusted environments (namely Try),
we recursively chown existing caches if there is a uid/gid mismatch.
We do this because Try is a sandbox and any random task could
experiment with a non-standard uid/gid. That populated cache would
"poison" the cache for the next caller. Or vice-versa. It would be
annoying if caches were randomly poisoned due to Try pushes that
didn't realize there was a UID/GID mismatch. We could outlaw "bad"
UID and GIDs. But that makes the barrier to testing things on Try
harder. So, we go with the flow and recursively chown caches in
this scenario.
This change will shine light on all tasks using inconsistent UID
and GID values on the same cache. Bustage is anticipated.
Unfortunately, we can't easily know what will break. So it will be
one of those things where we will have to fix problems as they arise.
Fortunately, because caches are now tied to the content of run-task,
we only need to back out this change and tasks should revert to caches
without UID and GID pinning requirements and everything will work
again.
MozReview-Commit-ID: 2ka4rOnnXIp
--HG--
extra : rebase_source : ccb2b0a9230694f989775b26d5276fd3ac928af3
extra : source : 083d2e1cc8fe44b04e44f74bda3dd8bc75ba826c
run-task's --chown and --chown-recursive are only used on volumes and
caches - the only locations that aren't controlled by the Docker image
itself and thus whose permissions could be "undefined."
Previous commits have taught run-task about the locations of all caches
and volumes. Therefore, we no longer need to manually define paths to
chown. Instead, we can chown as a side-effect of the path being a
cache or a volume.
So, this commit changes run-task to chown caches and volumes
automatically. Since we no longer have a use for --chown and
--chown-recursive, those arguments are removed.
There /could/ be some paths that are caches or volumes but aren't
getting defined as such in Taskgraph. I consider this a bug in
Taskgraph and the recourse is to properly define a path as a cache or
a volume there.
MozReview-Commit-ID: 1yqrhjil6gy
--HG--
extra : rebase_source : 82c9ccb33f09d3ce50a6b7c6021e1e5b77d1b738
We recently introduced support for telling run-task about caches so
it could sanitize them automatically. We also recently taught
docker-worker and docker-engine how to declare volumes.
Building on that work, we now pass a list of paths corresponding
to Docker volumes to run-task.
run-task now verifies volumes behave as expected. Unless the volume
paths correspond to caches, run-task verifies they are empty and chowns
them to an appropriate owner.
Requiring empty volumes is an arbitrary decision. But as the inline
comment says, it keeps things simpler and makes caches and volumes
behave more like each other.
MozReview-Commit-ID: 5lm2uIitrS3
--HG--
extra : rebase_source : abb0be1a36f24d8d818515885dd2a75d13aed48a
We're about to ban files in Docker volumes so they behave almost
identically to caches (which start empty).
We move the install of nexus.xml from Docker image time to
task time. This also means that changes to nexus.xml don't result
in having to rebuild the Docker image.
MozReview-Commit-ID: JIjeJN4mt2
--HG--
rename : taskcluster/docker/android-gradle-build/nexus.xml => taskcluster/scripts/builder/build-android-dependencies/nexus.xml
extra : rebase_source : 53848f06820bda7979b2ae15456e07f8aed2363d
See the inline comment for the rationale here.
This check may not catch all volumes and caches. But after subsequent
commits refactor how permissions for caches and volumes are handled,
this edge case will likely result in permissions errors in the task,
so it isn't worth worrying about.
Several Dockerfile have been updated to add missing VOLUME so the check
passes.
In the case of desktop1604-test, we stopped removing
/home/worker/.cache because you can't remove a mount point, which is
what volumes are inside Docker containers.
MozReview-Commit-ID: GEyNkkX00kN
--HG--
extra : rebase_source : 19b3165d8c6e4431aa43dca51828be255e82e34f
Docker volumes are host-mounted filesystems. We typically mount
caches at their location. But not always. The reason we define
VOLUME in Dockerfiles is we're guaranteed to get a fast host
filesystem instead of AUFS when a cache isn't mounted.
In this commit, we teach the docker-worker payload builder about
the existence of Docker volumes. Docker volumes can be declared
inline in the YAML. More conveniently, we automatically parse out
VOLUME lines from corresponding in-tree Dockerfile.
We'll do useful things with this data in subsequent commits.
MozReview-Commit-ID: BNxp8EDEYw
--HG--
extra : rebase_source : 1aa6dbb000386cd2789d526708ce369640d532c9
Previously, we conditionally added caches to a task if the current
parameters warranted it.
In order to audit that all caches fulfill basic requirements, we need
to have unconditional knowledge of all caches.
This commit introduces an optional key on each cache entry stating
whether it should be skipped in "untrusted" environments. When we
convert a task definition to a worker payload, we filter out these
caches if necessary.
This change uncovered an inconsistency with filtering caches. In
one location we filtered on the source repo name. In others, we
filtered on the SCM level.
Setting the caches in the spidermonkey kind also changed slightly
to ensure we're not overwriting existing caches. I don't think this
has any behavior changes. But the new method is more correct.
MozReview-Commit-ID: 1crpdWHqQ68
--HG--
extra : rebase_source : 69469014dcc92d4de98d5e887b10542af00e05be
~ in paths is mostly a shell-ism. Let's not use it.
The real reason for this is it interfes with upcoming commits that
audit cache and volume paths.
MozReview-Commit-ID: AhjMwg5gexx
--HG--
extra : rebase_source : dd997a5d9c1123beecaa14bc5458a9338b3e07db
Static analysis in my editor was complaining about the old pattern.
Why not fix it while I'm here.
MozReview-Commit-ID: HtrGenolNXb
--HG--
extra : rebase_source : e3ffc4f7961dfd90767b6d1af2b93ad7551de3be
This needs to be globally enabled to support sparse checkouts
and accessing repos using sparse checkouts.
Having the extension enabled should no-op unless sparse checkouts
are being used. i.e. it should be harmless to globally enable.
MozReview-Commit-ID: AKNUOXfYQPx
--HG--
extra : rebase_source : d32b8a89c56c39923d7b0cd61583c2828a29a872
We use Mercurial 4.3.1 pretty much everywhere in CI now.
Mozharness should be testing with it as well.
MozReview-Commit-ID: HT2rocEvdIe
--HG--
extra : rebase_source : 148a9cd82b18e693ee570f31fc961373e8466a3c
The GCC plugin used for hazard builds was built for GCC 4.9. Until a new
plugin is built by a toolchain job for the "default" version of GCC we
build Firefox with, the builds using the plugin need to stay with the
version the plugin was built for.
This corrects a bug introduced in 265873cf1388.
MozReview-Commit-ID: LkZlTVAM17E
--HG--
extra : rebase_source : bf442e620abd6b47adee3d4c56e6f0c19964aea1
We added this versioning as a mechanism to create clean breaks when
version control mechanisms change in significant ways. Now that
run-task's content influences cache names and this cache name
pattern must be used with run-task, we no longer need this
versioning and can rely on run-task's content and cache requirements
to switch to new caches.
MozReview-Commit-ID: KJ7IF7UzDsg
--HG--
extra : rebase_source : 9d562ec3977fea7920039a66be13cfae8980475e
run-task just grew features to aid with cache validation.
Attempts by run-task to use caches not under its control will fail.
So, we add a transform that audits for and ensures that certain
caches are only being used with run-task. This will help catch
stragglers attempting to use e.g. the legacy VCS checkouts or
tooltool caches without run-task. Fortunately, there are no
violations for this policy. Yay!
MozReview-Commit-ID: LBCmDUdgcuM
--HG--
extra : rebase_source : 9a290470088a8d55693dfbadc5186a018d3669b6
Today, cache names are mostly static and are brittle as a result.
In theory, when a backwards incompatible change is performed on
something that touches a cache, the cache name needs to be changed
to ensure tasks running the old code don't see cached data from the
new task. (Alternatively, all code is forward compatible, but that is
hard to implement in practice.)
For many things, the process works as planned. However, not everyone
knows that cache names need changed. And, it isn't always obvious
that some things require fresh caches. When mistakes are made, tasks
break intermittently due to cache wonkiness.
One area where we get into trouble is with UID and GID mismatch.
Task A will use a Docker image where our standard "worker" user/group
is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is
common when mixing Debian and RedHel based distros.) If they use the
same cache, then Task B needs to chown/chmod all files in the cache
or there could be a permissions problem. This is exactly why
run-task recursively chowns certain paths before dropping root
privileges.
Permissions setting in run-task solves permissions problems. But
it doesn't solve content incompatibility problems. For that, you
need to change cache names, not use caches, or blow away content
when incompatibilities are detected.
This commit starts the process of adding a little bit more coherence
to our caching story.
There are two main features in this commit:
1) Cache names tied to run-task content
2) Cache validation in run-task
Taskgraph now detects when a task is using caches with run-task. When
caches and run-task are both being used, the cache name is adjusted to
contain a hash of run-task's content. When run-task changes, the cache
name changes. So, changing run-task ensures that all caches from that point
forward are "clean." This frees run-task and any functionality related
to run-task (such as maintaining version control checkouts) from
having to maintain backwards or forwards compatibility with any other
version of run-task. This does mean that any changes to run-task
effectively wipe out caches. But changes to run-task tend to be
seldom, so this should be acceptable.
The second part of this change is code in run-task to record per-cache
properties and validate whether a populated cache is appropriate for
use. To enable this, taskgraph passes a list of cache paths via an
environment variable. For each cache path, run-task looks for a
well-defined file containing a list of "requirements." Right now,
that list is simply a version string. But other features will be
worked into it. If the cache is empty, we simply write out a new
requirements file and are done. If the file exists, we compare
requirements and fail fast if there is a mismatch. If the cache
has content but not this special file, then we abort (because this
should never happen).
The "requirements" validation isn't very useful now because the only
entry comes from run-task's source code and modifying run-task will
change the hash and cause a new cache to be used. The implementation
at this point is more demonstrating the concept than doing anything
terribly useful with it.
MozReview-Commit-ID: HtpXIc7OD1k
--HG--
extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
I must have been in a closure mood when I wrote this code. The
main function is getting a bit heavyweight. So let's extract
these closures to make things less dense.
MozReview-Commit-ID: 4p5yKB1tTxn
--HG--
extra : rebase_source : 3c3e0b352da6290043013aa36c783e21e01460ba
extra : source : 053f0b0b48635c6a87aefe15ad73f361f0f64f79
So we can use it from another module without feeling dirty.
MozReview-Commit-ID: 1TCk9dda6mL
--HG--
extra : rebase_source : 15c2ee0e4d6aa812ab8acc3564da0c3e7ba45e19
Caches shared across levels scare me, even if readers are purported to
perform content verification. We shouldn't take any risks with released
Firefox builds being contaminated by e.g. Try tasks.
Also, the old cache name interferes with my desire to make cache
names dynamic. This requires dynamic scopes. We already have
have level-{{level}}-* scopes for caches. So having all caches
prefixed with this makes things flexible.
MozReview-Commit-ID: LsrcxIYoEh1
--HG--
extra : rebase_source : dfe97f92a726059200ed79afe215ef2cf1fd7bf1
This pattern is repeated a number of times. Let's consolidate it.
A few places set relengapi-proxy=False. AFAICT this was a no-op because
the default value in the worker schema is False. So, these assignments
were dropped.
MozReview-Commit-ID: EoNPBhaMT2J
--HG--
extra : rebase_source : 72b9c6e4b42369720a19bab42c9ea40c942b5494
bug 1382280 tripled the size of desktop-build image because of
installation of debug symbols. It is only used for valgrind, so let's
move valgrind task to use its own image.
MozReview-Commit-ID: 16St7dDj8tr
--HG--
rename : taskcluster/docker/desktop-build/Dockerfile => taskcluster/docker/valgrind-build/Dockerfile
extra : rebase_source : cc66813cab430d906643fbadf63c661e14784f6f
This change adds an upload-generated-sources task kind that runs after nightly
builds, fetches their `target.generated-files.tar.gz` artifact, and uploads
all the contained files to an S3 bucket. For actual nightly and release builds
on SCM level 3 trees, the S3 bucket is configured to be publicly accessible,
so that tools like Socorro will be able to fetch generated source files that
appear in crash reports, and debuggers will be able to fetch generated sources
when they show up while debugging Nightly or Release builds.
There are also level-2 and level-1 S3 buckets configured for builds happening
on trees of other levels such as try. They are not configured as publicly
accessible, but they exist so that these tasks can be tested in try.
MozReview-Commit-ID: Js1HRftbtep
--HG--
extra : rebase_source : b1172c9cc8b8be437d3b94a6bf0ff6b2f7d3508b
extra : source : 73bf88110b3821d62a3d393e85b56896a12f2930
Set these as tier 3 right now, because there are frequent intermittent test failures.
MozReview-Commit-ID: Gs6AXLKaVrF
--HG--
extra : rebase_source : b86c6ffd527dc8af963967c2722a3cee66a6ed3f
Gregory Szorc
Wes Kocher
Andrew Halberstadt
J. Ryan Stinnett
Dustin J. Mitchell
Mike Hommey
Gregory Arndt
Aki Sasaki
Rok Garbas
Sebastian Hengst
Justin Wood
Wander Lairson Costa
Ryan VanderMeulen
Tom Prince
Brian Stack
Phil Ringnalda
Ted Mielczarek