Without this, "debug" and "opt" builds share the same cache. This
causes builds to randomly fail in automation.
MozReview-Commit-ID: Fend6GGmhFk
--HG--
extra : rebase_source : c3370938364b1e2cc94081d58ba9f35ce1060e5d
This fixes the use of the bogus pattern `taskcluster/**.py` which does not
actually match python files in subdirectories.
MozReview-Commit-ID: 3UqM5UQ0HOE
--HG--
extra : rebase_source : 73eec18034a11256273589b69b6d95aabceecd03
We've created a new worker type based on c4.4xlarge and m4.4xlarge
instances to replace the existing worker types based on c3.2xlarge,
m3.2xlarge, and r3.2xlarge. The new worker type has 16 vCPU instead
of 8, which makes build tasks complete ~10 minutes faster.
We've also consolidated down into a single pool for workers.
MozReview-Commit-ID: sOfKspFjCS
--HG--
extra : rebase_source : 8c7b030e0d9376dd9c07b71b8987be0c300fb691
This was recently refactored (incorrectly) in 147b245de122
(bug 1290282). Before that refactor, we only defined {{build_name}}
in base_linux64.yml. After, we hard-coded "linux32" and "linux64."
The change to linux64 resulted in ASAN builds sharing the same cache
name. Derp.
This commit restores {{build_name}} to the cache path. And for
consistency, we use {{build_name}} on both 32 and 64-bit variants.
MozReview-Commit-ID: KhqIjzARETO
--HG--
extra : rebase_source : 6b110302b66e052ec3d79bdc5b05a190cb0a7749
We run only a small subset of the test suite to save resources. Over
time, we'll run more of the platform test suite.
--HG--
extra : rebase_source : 264fbe55be5b56857278ae8d731ef51b1b41cf3c
extra : histedit_source : a3550c7e2ea8a41cd6285e511b7268a9e7af11cd%2Cec6024dadcb7c071170aa452064776666283bb27
Before, we simply executed scripts inside Docker containers. This
frequently resulted in a wall of text with command output. It was
difficult to discern things like the time spent performing certain
actions.
Before, individual tasks had to drop permissions from the default
root user themselves. Dropping permissions isn't exactly a trivial
thing to do and a number of tasks didn't do it or did it wrong.
Before, we had a "checkout-gecko-and-run" script that kinda/sorta
did common activities for us. But it was written as a shell script
and doing advanced things was difficult.
This commit can be treated as a rewrite of "checkout-gecko-and-run"
as a Python script. But it also does a bit more. It prefixes output
with timestamps so we know how long operations took. It features more
robust argument parsing, so we can add new features more easily.
To prove the new wrapper script works, the lint image and all tasks
using it have been converted to use it.
MozReview-Commit-ID: 5d95u5Xebtq
--HG--
extra : rebase_source : 3a1d84782b01d7743e846bd0c04d7867813dd8a3
Currently, TaskCluster tasks tend to use the "workspace" directory as
a cache that manages the source checkout *and* additional state.
Historically at Mozilla, we've lumped "source checkout" and "workspace"
(sometimes known as an "objdir") into the same directory. This is
not ideal. Ideally, there is an immutable, read-only source checkout
and all files produced from that source live in a separate directory.
In this commit, the "workspace" directory for the "lint" image has been
renamed to "checkouts" and all tasks using the image have been updated
accordingly. By having "checkout" in the name, we clearly identify this
cache as being relevant to source checkouts, which IMO can serve a
different role from "workspaces." This distinction is important, as the
next commit will prevent the "checkouts" cache from getting optimized
out in certain tasks.
To hammer this point home, documentation on common caches has been
introduced.
MozReview-Commit-ID: BSEc4dM5YCt
--HG--
extra : rebase_source : 5a62939e066d3723736b41e14007112d92346684
DONTBUILD because it's not worth it.
--HG--
extra : rebase_source : 1cea5d6121fda41fc668bf24990734abf4a56677
extra : amend_source : 318ec9edcf8072ffe1840b5f7d8ce144be63424f
This removes the in-tree plugin from the tooltool archive and uses that code
directly from the Gecko checkout instead.
For automation, we now get ESLint and external plugins from tooltool and then
symbolic link to the in-tree plugin.
For local development, we install ESLint and external plugins following the
shrinkwrap file created from the last change to the tooltool archive. The local
plugin is then installed.
This change also removes the list of module versions from mach_commands.py, so
there is only one place to update module versions for the future.
MozReview-Commit-ID: AhbZ8lVPmN4
The first thing we do is switch from the desktop-test image to the
lint image: the mozharness tests don't need anything in the
desktop-test image. This does mean converting from Ubuntu 12.04
to 16.04. I'm fine upgrading to a newer operating system by
4 years :)
Once we're on the lint image, we switch to use checkout-gecko-and-run.
The mozharness-gecko task now does version control just like
eslint-gecko and mozlint-flake8.
MozReview-Commit-ID: 6mzAZ7LSBWl
--HG--
extra : rebase_source : a766e5db61998d1cfbcb792ad09a265f45027f54
Previously, every lint task would have to create its own checkout. This
was time consuming.
The robustcheckout extension purges the working copy of *all*
untracked and ignored files. It also restores modified files to
their original state. In other words, as long as you trust Mercurial
to go from revision X to revision Y, robustcheckout is as good as
a fresh checkout.
This commit adds a cache for the working directory checkout so lint
tasks only have to effectively perform incremental `hg update` between
task executions. This should make tasks spend a lot less time
doing version control foo.
On Try, time for flake8 tasks is currently hovering around 4 minutes.
After this change, I've seen tasks finish as quickly as 11s! But that
was with a hacked up legacy.py that added the workspace cache to the
whitelist for Try. While I would like to see workspace reuse on Try
eventually, this is not the right commit to roll that out in.
MozReview-Commit-ID: 66P2rt896qE
--HG--
extra : rebase_source : 2a7c8d396e85ba4eae84f8843256050a2288c9d0
extra : intermediate-source : c478ecb14bf6164ef0c955acee3a0a2f18e415c5
extra : source : 63d54efdc1f8effb0370644c11014a3f0404073b
extra : histedit_source : 560701ce41c870171443c78e3a9de3998fbf9306
The robustcheckout Mercurial extension does a clone+checkout optimally.
Read the bug for more on it.
robustcheckout is already used by mozharness automation. It has resulted
in a significant reduction in I/O usage and utilization in automation.
This commit replaces tc-vcs with the robustcheckout equivalent.
We replace the existing tc-vcs scope and cache with a new one.
Because Dustin and I are paranoid, we maintain separate caches per
SCM level - even though we could arguably share the same cache. Defense
in depth.
Robustcheckout (when used with --sharebase) pools storage for related
repos automatically. i.e. changesets from inbound and central will
be in the same store. This means you likely only have one copy of
each changeset per cache. This can result in significant space savings.
And, since there are fewer copies floating around, hg.mozilla.org
and various network appliances are working less too!
Since tc-vcs is no longer used, we stop it from being installed.
While we're here, we also change the images to execute as the
"worker" user. This happens automatically as a result of using
the "checkout-and-run" script.
MozReview-Commit-ID: EDeebuP7TkT
--HG--
extra : rebase_source : 2bec5dd9d6fe5565831bb35f195859aa12dd0bf2
extra : intermediate-source : 06481d97a485f6566554b087bc3880d76361e8ec
extra : source : d368700c93ef085325a081219d7aeb8512bc54a1
extra : histedit_source : c07505273fc8f10acf8e8d3ee01e327afd0aa63d
Wes Kocher
Geoff Brown
aselagea
Joel Maher
Steve Fink
Kim Moir
Carsten "Tomcat" Book
Dustin J. Mitchell
Gregory Szorc
Nick Alexander
James Graham
Henrik Skupin
J. Ryan Stinnett
Rob Thijssen
Justin Wood