This creates a new "job-from" field that contains the relative filename the job was defined
in. The filename is relative to 'config.path'. If the task came from the 'jobs' key defined
in kind.yml, this field will be set to 'kind.yml'.
MozReview-Commit-ID: 9e1tEb6XuZT
--HG--
rename : taskcluster/ci/source-test/python-tests.yml => taskcluster/ci/source-test/python.yml
extra : rebase_source : 89010f00a96f88a4eecc8b37028f26b483b820ec
Clean up and standardize Treeherder symbols for Talos and AWSY tasks:
* Stylo disabled groups include `sd`
* Stylo sequential groups include `ss`
MozReview-Commit-ID: 7cl6e0XvXNO
--HG--
extra : rebase_source : a7b4e7c78af673e204d7ff7479ce5fc87614248c
Convert all jobs that were exercising Stylo enabled to Stylo disabled instead.
Stylo enabled is now handled by the default jobs.
In Perfherder, Stylo enabled jobs will be untagged and take over the existing
Gecko series. Stylo disabled jobs will have a new `stylo-disabled` tag and
create a new series.
MozReview-Commit-ID: BMXBRg3A95j
--HG--
extra : rebase_source : 0f3a6a1f6807dbb90c00d060a3a7f220c2141c64
The signing pool is a limited resource. We already have logic to avoid re-signing
widevine if the .sig files are there, and we're taking advantage of that on linux
and windows. OSX was the lone desktop os still pointing at the unsigned bits for
l10n.
We're pointing at repackage instead of signing because the l10n logic expects a dmg.
MozReview-Commit-ID: FtWa4o5s7aJ
--HG--
extra : rebase_source : f704a97293a231d26a4806543f53fc7e83eb9ca9
This means that a push to try affecting only Android will only run android builds
and tests, for example.
MozReview-Commit-ID: HVUvIg0EUZn
--HG--
extra : rebase_source : 5007d3723cd863e4e98fdbdd874e5ba70b44ba0c
This adds some new optimization strategies. For tests, we use Either(SETA,
SkipUnlessSchedules), thereby giving both mechanisms a chance to skip tasks. On
try, SETA is omitted.
MozReview-Commit-ID: GL4tlwyeBa6
--HG--
extra : rebase_source : 0c1ce762afc7a691788379d4f4206df106f6df63
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.
The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.
MozReview-Commit-ID: AXJEGLh6pEV
--HG--
extra : rebase_source : fdd8c3cfb9abf759a3c43c5713e62e4772c5bd06
In preparation for much more thorough optimization of task-graphs, this
makes a few changes:
* optimization is split into thre phases, with task removal in one phase
(following dependency links) and task replacement in the next (in the
reverse order).
* optimization uses class instances instead of functions for optimizations;
this allows different functions for different phases, and also leaves open
the possibility of composing optimizations.
* the replacement phase can also support removal; this is when utility tasks
like symbol uploads can be optimized away iff their parent task is
optimized.
MozReview-Commit-ID: C5QznNpwqXn
--HG--
extra : rebase_source : f8e65ee3ebb9fb584b00ee0db518b790a9c1b233
It is not at *all* clear how multiple optimizations for a single task should
interact. No simple logical operation is right in all cases, and in fact in
most imaginable cases the desired behavior turns out to be independent of all
but one of the optimizations. For example, given both `seta` and
`skip-unless-files-changed` optimizations, if SETA says to skip a test, it is
low value and should be skipped regardless of what files have changed. But if
SETA says to run a test, then it has likely been skipped in previous pushes, so
it should be run regardless of what has changed in this push.
This also adds a bit more output about optimization, that may be useful for
anyone wondering why a particular job didn't run.
MozReview-Commit-ID: 3OsvRnWjai4
--HG--
extra : rebase_source : 4ac6a5cc592f4210918c73e667f3b5dd50230894
This includes adding TASKCLUSTER_VOLUMES to docker image builds directly. The
env variable is not added as part of the task transform because `run-task` is
not in payload.command. In fact, build-image.sh calls run-task after doing
some other housekeeping.
Ideally image builds would be turned into jobs and all of this would occur
automatically, but that turns out to be quite a bit too complex for this
incidental fix -- perhaps best solved in another bug.
MozReview-Commit-ID: FYHvafJras7
--HG--
extra : rebase_source : 4e3b9ae9900727e7932c13ced34b3f8596d755d9
This adds a new morph template for modifying a task's env and the corresponding glue
to specify it from |mach try fuzzy|. It can be used like:
./mach try fuzzy --env FOO=1 --env BAR=baz
This will simply set those environment variables in *all* tasks. We could add the
ability to only specify it for a subset of tasks in the future, but that seems like
a hard problem that probably isn't worth it.
MozReview-Commit-ID: C4sokv886PU
--HG--
extra : rebase_source : 64c9105848e6becd33fae63aeabc3423cfda3bdf
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: 554IPMRWgzK
--HG--
extra : rebase_source : 00827d3f6bd705419bc801eb05b543af1ddc274f
This fixes the issue where "-u web-platform-tests-wdspec" scheduled
nothing. Now it will schedule a e10s-enabled run of wdspec tests.
MozReview-Commit-ID: AdHZqtk1hLy
--HG--
extra : rebase_source : 8d5926277b37952616c4dffdb20133868913bab2
`run-task` is taught a --sparse-profile argument to be passed down
to `hg robustcheckout` for the main source checkout. It does what
you expect: performs a sparse checkout using the named profile.
The Taskgraph YAML for run-task is taught a "sparse-profile"
property to define the sparse profile. When defined, --sparse-profile
will be passed down to `run-task` and the cache name will be updated
to reflect the use of sparse checkout.
Our cache checking transform is updated to audit for the use of
--sparse-profile without the corresponding "-sparse" cache name
variation.
The reason we need a distinct cache name for sparse is because
clients that aren't sparse aware will be unable to read checkouts
that are sparse. By forcing sparse and non-sparse into different
cache pools, we avoid compatibility issues.
In the ideal world, we probably support sparse profiles on all the
VCS checkouts that `run-task` supports (e.g. --tools-checkout).
Perfect is the enemy of done. All of this is defined in-tree and
it is easy enough to change atomically.
MozReview-Commit-ID: 79k7Vul0hHO
--HG--
extra : rebase_source : babe9b42e2796c2341bffc6ecfe829f4daff9e0f
* eliminate heading for test kinds, of which there is now only one
* make the caches document have a single heading in the TOC
* break out mach commands into a separate document, add ./mach taskgraph morphed
* remove docs for YAML templates support (the .yml file wasn't actually
used -- I expect it was a merge leftover); these are still used for actions.yml,
but once that is gone the code should be removed, too.
* break try out into its own document, edit to distinguish "how to run try"
from "how to generate config"
MozReview-Commit-ID: 76ZopWA9TPL
--HG--
extra : rebase_source : 6946d866f9df6eec591b9a05ddedc6467dd69e4b
The UID and GID that a task executes under is dynamic. As a result,
caches need to be aware of the UID and GID that owns files otherwise
subsequent tasks could run into permission denied errors. This is
why `run-task --chown-recursive` exists. By recursively changing
ownership of persisted files, we ensure the current task is able
to read and write all existing files.
When you take a step back, you realize that chowning of cached
files is an expensive workaround. Yes, this results in cache hits.
But the cost is you potentially have to perform hundreds of thousands
of I/O system calls to mass chown. The ideal situation is that
UID/GID is consistent across tasks on any given cache and
potentially expensive permissions setting can be avoided. So, that's
what this commit does.
We add the task's UID and GID to run-task's requirements. When we
first see a cache, we record a UID and GID with it and chown the
empty cache directory to that UID and GID. Subsequent tasks using
this cache *must* use the same UID and GID or else run-task will
fail.
Since run-task now guarantees that all cache consumers use the same
UID and GID, we can avoid a potentially expensive recursive chown.
But there is an exception. In untrusted environments (namely Try),
we recursively chown existing caches if there is a uid/gid mismatch.
We do this because Try is a sandbox and any random task could
experiment with a non-standard uid/gid. That populated cache would
"poison" the cache for the next caller. Or vice-versa. It would be
annoying if caches were randomly poisoned due to Try pushes that
didn't realize there was a UID/GID mismatch. We could outlaw "bad"
UID and GIDs. But that makes the barrier to testing things on Try
harder. So, we go with the flow and recursively chown caches in
this scenario.
This change will shine light on all tasks using inconsistent UID
and GID values on the same cache. Bustage is anticipated.
Unfortunately, we can't easily know what will break. So it will be
one of those things where we will have to fix problems as they arise.
Fortunately, because caches are now tied to the content of run-task,
we only need to back out this change and tasks should revert to caches
without UID and GID pinning requirements and everything will work
again.
MozReview-Commit-ID: 2ka4rOnnXIp
--HG--
extra : rebase_source : ccb2b0a9230694f989775b26d5276fd3ac928af3
extra : source : 083d2e1cc8fe44b04e44f74bda3dd8bc75ba826c
run-task's --chown and --chown-recursive are only used on volumes and
caches - the only locations that aren't controlled by the Docker image
itself and thus whose permissions could be "undefined."
Previous commits have taught run-task about the locations of all caches
and volumes. Therefore, we no longer need to manually define paths to
chown. Instead, we can chown as a side-effect of the path being a
cache or a volume.
So, this commit changes run-task to chown caches and volumes
automatically. Since we no longer have a use for --chown and
--chown-recursive, those arguments are removed.
There /could/ be some paths that are caches or volumes but aren't
getting defined as such in Taskgraph. I consider this a bug in
Taskgraph and the recourse is to properly define a path as a cache or
a volume there.
MozReview-Commit-ID: 1yqrhjil6gy
--HG--
extra : rebase_source : 82c9ccb33f09d3ce50a6b7c6021e1e5b77d1b738
We recently introduced support for telling run-task about caches so
it could sanitize them automatically. We also recently taught
docker-worker and docker-engine how to declare volumes.
Building on that work, we now pass a list of paths corresponding
to Docker volumes to run-task.
run-task now verifies volumes behave as expected. Unless the volume
paths correspond to caches, run-task verifies they are empty and chowns
them to an appropriate owner.
Requiring empty volumes is an arbitrary decision. But as the inline
comment says, it keeps things simpler and makes caches and volumes
behave more like each other.
MozReview-Commit-ID: 5lm2uIitrS3
--HG--
extra : rebase_source : abb0be1a36f24d8d818515885dd2a75d13aed48a
See the inline comment for the rationale here.
This check may not catch all volumes and caches. But after subsequent
commits refactor how permissions for caches and volumes are handled,
this edge case will likely result in permissions errors in the task,
so it isn't worth worrying about.
Several Dockerfile have been updated to add missing VOLUME so the check
passes.
In the case of desktop1604-test, we stopped removing
/home/worker/.cache because you can't remove a mount point, which is
what volumes are inside Docker containers.
MozReview-Commit-ID: GEyNkkX00kN
--HG--
extra : rebase_source : 19b3165d8c6e4431aa43dca51828be255e82e34f
Docker volumes are host-mounted filesystems. We typically mount
caches at their location. But not always. The reason we define
VOLUME in Dockerfiles is we're guaranteed to get a fast host
filesystem instead of AUFS when a cache isn't mounted.
In this commit, we teach the docker-worker payload builder about
the existence of Docker volumes. Docker volumes can be declared
inline in the YAML. More conveniently, we automatically parse out
VOLUME lines from corresponding in-tree Dockerfile.
We'll do useful things with this data in subsequent commits.
MozReview-Commit-ID: BNxp8EDEYw
--HG--
extra : rebase_source : 1aa6dbb000386cd2789d526708ce369640d532c9
Previously, we conditionally added caches to a task if the current
parameters warranted it.
In order to audit that all caches fulfill basic requirements, we need
to have unconditional knowledge of all caches.
This commit introduces an optional key on each cache entry stating
whether it should be skipped in "untrusted" environments. When we
convert a task definition to a worker payload, we filter out these
caches if necessary.
This change uncovered an inconsistency with filtering caches. In
one location we filtered on the source repo name. In others, we
filtered on the SCM level.
Setting the caches in the spidermonkey kind also changed slightly
to ensure we're not overwriting existing caches. I don't think this
has any behavior changes. But the new method is more correct.
MozReview-Commit-ID: 1crpdWHqQ68
--HG--
extra : rebase_source : 69469014dcc92d4de98d5e887b10542af00e05be
~ in paths is mostly a shell-ism. Let's not use it.
The real reason for this is it interfes with upcoming commits that
audit cache and volume paths.
MozReview-Commit-ID: AhjMwg5gexx
--HG--
extra : rebase_source : dd997a5d9c1123beecaa14bc5458a9338b3e07db
Static analysis in my editor was complaining about the old pattern.
Why not fix it while I'm here.
MozReview-Commit-ID: HtrGenolNXb
--HG--
extra : rebase_source : e3ffc4f7961dfd90767b6d1af2b93ad7551de3be
Joel Maher
Dustin J. Mitchell
J. Ryan Stinnett
Andrew Halberstadt
Sebastian Hengst
Nick Fitzgerald
Aki Sasaki
Ryan VanderMeulen
Phil Ringnalda
Wes Kocher
Johan Lorenzo
Brian Stack
Mike Hommey
Wander Lairson Costa
Maja Frydrychowicz
Gregory Szorc