This commit does a lot. But it's really not too difficult to comprehend
once you focus on the final state, which is basically the same as the
"lint" image and derived tasks.
Before, the "decision" image contained a "checkout-gecko" script and
"run-action" and "run-decision" scripts. The latter 2 invoked the first
script.
The "run-task" script basically does what the combination of these
scripts were doing before. So we switch to it.
While we're here, we also replaced the custom Mercurial installation in
this image with the shared install-mercurial.sh script. The
system-setup.sh script for the decision image is now short and sweet.
The YAML files for tasks using this image have been updated to use
"run-task." We no longer have to pass an environment variable to hold
command arguments. So we revert to putting these arguments inline in
the task's command. Dustin never liked passing the arguments as
environment variables, so it should make him happy ;) We add
--log-no-times because "run-task" prefixes its own timestamps on output
lines.
The path to the Gecko checkout has been changed from
/home/worker/workspace to /home/worker/checkouts to match changes made
in bug 1289643.
Finally, since "checkout-gecko-and-run" is no longer used, we delete it.
The Docker image version has been bumped accordingly.
MozReview-Commit-ID: HAGuCz6668Q
--HG--
extra : rebase_source : 9586873c21d67f8ce869b7ceee221e8fa8775539
The reason tooltool.py was vendored in testing/docker/decision was
because locally built Docker images were using vanilla `docker build`
and didn't know about our special Dockerfile syntax to allow the
inclusion of images from outside the directory where the Dockerfile
was located.
Now that locally-built Docker images know of our special Dockerfile
syntax, we can include files from anywhere. So, move tooltool.py
to a shared directory, away from the decision image.
I didn't bump the version of the decision image because there are
a few more things I want to do to this image, such as have it use
the `checkout-gecko-and-run` script instead of its own script.
I think I'll do that in a separate bug, however.
MozReview-Commit-ID: 12ze9GJFMJq
--HG--
rename : testing/docker/decision/tooltool.py => testing/docker/recipes/tooltool.py
extra : rebase_source : cfc2fa5759a3ec708b6e1736e9a2da03f0d25428
Changes to the decision Docker image have been compelted. We're ready to
use the new image.
We tag the image, update version references, change the task caches
so the new Mercurial pooled storage from the robustcheckout extension is
used, and convert the decision tasks to run as the "worker" user.
MozReview-Commit-ID: 61v9Ivy59zG
--HG--
extra : rebase_source : 640318a87660950c5e0680867a1bfdd68e35f127
extra : histedit_source : ec53fc576c00e5f2053167b37544ac7afccaecb5
When we switch to use robustcheckout for version control foo, we'll
also be taking the opportunity to have the decision and action tasks
execute as the "worker" user.
Since caches are mounted and owned by root and since tasks initially
run as root, this makes defining the container command in YAML a bit
difficult because we have to do some work as root then switch users
and continue executing. Rather than shoehorning all that complicated
logic into YAML, we introduce bash scripts that do it. These will
be plugged into the task YAML when we formally switch the tasks
to use the new Docker image.
We provide one script for running Gecko decision tasks. We provide
another for running action tasks. These are the two consumers of
the decision image we care about.
We also sneak in a change to add the executable bit to checkout-gecko.
MozReview-Commit-ID: CXlyHZJSHcP
--HG--
extra : rebase_source : 80621d4833a9d745eaff7da4641dfd4ace8ae1db
extra : histedit_source : e6ce7de5d14c8781d8dd94a8eff76c3227cd18b5
Now that Mercurial 3.8.4 and robustcheckout are in place, we convert
checkout-gecko from tc-vcs to robustcheckout.
As part of this, we remove references to tc-vcs from the Docker image.
This completes our changes to the decision Docker image. Image size has
been reduced from ~725 MB to ~217 MB. Not bad.
MozReview-Commit-ID: Hx9d02Al1TP
--HG--
extra : rebase_source : 05114e4e0e7fbbab2c89f25074abfeb7b9ba62ef
extra : histedit_source : 193c0bbb64cc1e468b5d7bb969d7f74e25947bde
web.cacerts matches what the Ubuntu package does by default.
[progress] changes are to make output in TaskCluster logs less
spammy (only 1 update per second instead of up to 10).
The robustcheckout extension will be used in a subsequent commit to
handle repository checkouts.
MozReview-Commit-ID: 2PvW4wEGk2u
--HG--
extra : rebase_source : 742627ba823d4f2097a4273e6cc6af8bb842c69f
extra : histedit_source : d479c1923c71605e9511e877b4b90d3b4d42f542
Previously, we were downloading tooltool.py from github.com. There
were a few problems with this.
First, there is a dependency on a 3rd party service. While the Docker
image should be cached, as a matter of principle we don't like hitting
3rd party services in our automation. The file is small enough, so we
just vendor it.
Second - and more importantly - we weren't validating the integrity of
the downloaded file. This means that a MiTM could possibly alter the
content of the file without us knowing (they would need a valid CA but
since the Ubuntu trusted CA bundle contains a lot of CAs from e.g.
governments, this isn't out of the question). Vendoring the file removes
this risk.
Third, behavior wasn't deterministic over time. We were always
downloading the "master" revision of the file. I like determinism over
time. Vendoring makes things deterministic.
MozReview-Commit-ID: 4DdSd42BnAu
--HG--
extra : rebase_source : cf73d2741fc186bebf06233efefdf85cd8cea3f2
extra : histedit_source : 76c7d81266a72010a9969ea32ac13c7bce2a0601
I'm not sure why the decision image has so many packages installed.
Most of them don't need to exist because the decision image only
needs to obtain a copy of the Firefox repo and run `mach`. This
doesn't require any build system per se. And all the Python
dependencies are vendored in the Firefox repo. All we need is a
Python 2.7 interpreter.
This change reduces the decision image size from ~700 MB to ~300 MB.
MozReview-Commit-ID: CUqc5TUVZSc
--HG--
extra : rebase_source : 5a2b3888b4c54c29bc8c8b9215ce36a4340574e5
extra : histedit_source : 61e70b06b703c3262ae1bc2f527f1919a3f450ec
We change the installation of Mercurial from via peep to .deb files in
tooltool. The .deb files were produced by Mercurial's built-in make
targets to produce .deb packages.
As part of this, we upgrade to Mercurial 3.8.4. It should be a drop-in
replacement.
Since we no longer use peep, we stop installing it and pip/setuptools
since they were only needed to run peep.
It's worth noting that we choose to install from .deb files instead of
pip because this keeps image creation small and simple. Otherwise we'd
have to install a compiler, etc.
MozReview-Commit-ID: INnKDHkX2uk
--HG--
extra : rebase_source : 0c6f30ff193dba5fbb5d90603e00f8be02816f9d
extra : histedit_source : 2afd18a694447bd133c26b7ccd562cdf7453b674
We're currently running Ubuntu 14.04 in the decision image. While it is
still in LTS support, 16.04 ships with a modern, properly configured Python
2.7. So we upgrade to 16.04 and drop the install of Python from source
because it is no longer needed.
This is part 1 of a larger refactor to this image.
MozReview-Commit-ID: CTbsPmTjcgs
--HG--
extra : rebase_source : eca12e98c8ff63cb302ea580da9296bd4cf31a4f
extra : histedit_source : 1a40405a9360239bf95d368c43ccfd0681609500
In preparation for running tasks as the worker user.
MozReview-Commit-ID: DLgD0lh5V2C
--HG--
extra : rebase_source : 1508517f9fbc986ada96cbe4ee77847ad6e1afcc
extra : histedit_source : 4b2957c47fcab8704416748613e7ff5badc61897
When cloning from mercurial the host fingerprint should be validated to the
currently known good fingerprint. This needs to be stored in a commit
independent way so that rerunning tasks in the future do not fail if
the mercurial server's fingerprint is different at that time.
MozReview-Commit-ID: GTermkwGgdb
--HG--
extra : rebase_source : 12bfaed59ac56a6cf81d7a738764a13161927c91
This version of tc-vcs fixes downloading of remote refs when they don't
exist locally [1].
[1] https://github.com/taskcluster/taskcluster-vcs/pull/12
--HG--
extra : histedit_source : e9a952d7b77d45bcb803d5a3188b93d3506c140c
Gregory Szorc
Gregory Arndt
Wander Lairson Costa
James Lal