Until now Gecko decision task artifacts would expire after a week. Unfortunately, some of these artifacts are
needed in order for Treeherder to determine which tasks it can schedule on a given push.
In this change, we're changing the expiration of all artifacts except those that are related to docker image
optimizations, which, are very large files and we only want to keep for a week.
MozReview-Commit-ID: 96ahyWmoI7W
--HG--
extra : rebase_source : d0f4d65e3b5bfae832a5aaed60c0dfd58f0bdc62
This commit does a lot. But it's really not too difficult to comprehend
once you focus on the final state, which is basically the same as the
"lint" image and derived tasks.
Before, the "decision" image contained a "checkout-gecko" script and
"run-action" and "run-decision" scripts. The latter 2 invoked the first
script.
The "run-task" script basically does what the combination of these
scripts were doing before. So we switch to it.
While we're here, we also replaced the custom Mercurial installation in
this image with the shared install-mercurial.sh script. The
system-setup.sh script for the decision image is now short and sweet.
The YAML files for tasks using this image have been updated to use
"run-task." We no longer have to pass an environment variable to hold
command arguments. So we revert to putting these arguments inline in
the task's command. Dustin never liked passing the arguments as
environment variables, so it should make him happy ;) We add
--log-no-times because "run-task" prefixes its own timestamps on output
lines.
The path to the Gecko checkout has been changed from
/home/worker/workspace to /home/worker/checkouts to match changes made
in bug 1289643.
Finally, since "checkout-gecko-and-run" is no longer used, we delete it.
The Docker image version has been bumped accordingly.
MozReview-Commit-ID: HAGuCz6668Q
--HG--
extra : rebase_source : 9586873c21d67f8ce869b7ceee221e8fa8775539
Currently, the gecko-decision and action tasks maintain a separate cache
for each "project" (Firefox repo) source checkout. In my opinion, this
is not necessary.
Separate caches/source checkouts mean a lower chance for a cache hit:
a lower chance that any given worker will have a source checkout readily
available for the "project" being built. They also mean workers store
more data and files in caches.
By using a shared cache for source checkouts, workers will ideally
only have a single source checkout cache and will be pretty much
guaranteed to have a checkout readily available (the main exception
being the first task on a freshly-provisioned worker).
A downside of a shared cache is that workers may spend more time doing
`hg update` operations. e.g. if the last task was from mozilla-release
and the current task is from mozilla-central, thousands of files will
be updated. However, I believe we will still be better off because
separate caches *increase* the probability for a full checkout, which
involves writing ~150,000 files! So I believe this change will lower
the overall amount of VCS work done by workers while possibly
sacrificing the lowest possible execution time (which is achieved by
minimizing the `hg update` work).
It's worth noting that `hg robustcheckout` (which is used by these
tasks) does the equivalent of an `hg purge --all`. This ensures that all
untracked and ignored files from previous tasks are deleted. So there
should be no contamination from previous tasks.
MozReview-Commit-ID: I5Ng3xFEfou
--HG--
extra : rebase_source : db423e5005d1e26d7dfebbbfa28893fc32caef6f
https://hg.mozilla.org/mozilla-unified contains heads from all the
major Firefox repos. In addition, it is encoded on the server in such a
way that it is several hundred megabytes smaller despite containing
30,000+ more changesets. This means faster clones and faster
operations.
Cloning this repo does require a new version of Mercurial. So if any
TC tasks are using an ancient Mercurial - one that is vulnerable to
known CVE issues in fact - this will flush them out.
MozReview-Commit-ID: 2VHDa6FEeeJ
--HG--
extra : rebase_source : db4990bcde0503fd14d82a5d16c71adbb4f92be3
extra : histedit_source : 6c555bd3df12536d1c48b45d8cc76611e3f7032f
Changes to the decision Docker image have been compelted. We're ready to
use the new image.
We tag the image, update version references, change the task caches
so the new Mercurial pooled storage from the robustcheckout extension is
used, and convert the decision tasks to run as the "worker" user.
MozReview-Commit-ID: 61v9Ivy59zG
--HG--
extra : rebase_source : 640318a87660950c5e0680867a1bfdd68e35f127
extra : histedit_source : ec53fc576c00e5f2053167b37544ac7afccaecb5
The decision task configures a /home/worker/workspace cache. However,
the command we run in the container references a "workspace" relative
path. From logs in automation, it appears that PWD during execution
is "/" because "workspace" is being resolved to "/workspace." The
net result of this is we appear to be performing a VCS clone+checkout
on every single task.
This commit fixes the paths so our workspace cache is actually used.
MozReview-Commit-ID: Kj6REep5bSs
--HG--
extra : rebase_source : 8cd8be43dfd34f2970b47721c3da8e3957a8bfed
extra : histedit_source : b81dce523a88e44eb3fa8b1a68840066edca382d
Jobs reporting to treeherder should rely on the task route for project,
revision, and pushlog ID rather than things stuffed into task.extra.treeherder.
This also removes the need for a revision_hash that was calculated by mozilla-taskcluster.
MozReview-Commit-ID: EcQM9QRZzgG
--HG--
extra : rebase_source : f04f6724feef2dd51b4b98c67c9a261b093f452b
extra : amend_source : 0590605834d93359206f49edd94396c43b57f6dd
Dustin J. Mitchell
Gregory Szorc
Armen Zambrano Gasparnian
Gregory Arndt