Before, we simply executed scripts inside Docker containers. This
frequently resulted in a wall of text with command output. It was
difficult to discern things like the time spent performing certain
actions.
Before, individual tasks had to drop permissions from the default
root user themselves. Dropping permissions isn't exactly a trivial
thing to do and a number of tasks didn't do it or did it wrong.
Before, we had a "checkout-gecko-and-run" script that kinda/sorta
did common activities for us. But it was written as a shell script
and doing advanced things was difficult.
This commit can be treated as a rewrite of "checkout-gecko-and-run"
as a Python script. But it also does a bit more. It prefixes output
with timestamps so we know how long operations took. It features more
robust argument parsing, so we can add new features more easily.
To prove the new wrapper script works, the lint image and all tasks
using it have been converted to use it.
MozReview-Commit-ID: 5d95u5Xebtq
--HG--
extra : rebase_source : 3a1d84782b01d7743e846bd0c04d7867813dd8a3
The reason tooltool.py was vendored in testing/docker/decision was
because locally built Docker images were using vanilla `docker build`
and didn't know about our special Dockerfile syntax to allow the
inclusion of images from outside the directory where the Dockerfile
was located.
Now that locally-built Docker images know of our special Dockerfile
syntax, we can include files from anywhere. So, move tooltool.py
to a shared directory, away from the decision image.
I didn't bump the version of the decision image because there are
a few more things I want to do to this image, such as have it use
the `checkout-gecko-and-run` script instead of its own script.
I think I'll do that in a separate bug, however.
MozReview-Commit-ID: 12ze9GJFMJq
--HG--
rename : testing/docker/decision/tooltool.py => testing/docker/recipes/tooltool.py
extra : rebase_source : cfc2fa5759a3ec708b6e1736e9a2da03f0d25428
The robustcheckout Mercurial extension does a clone+checkout optimally.
Read the bug for more on it.
robustcheckout is already used by mozharness automation. It has resulted
in a significant reduction in I/O usage and utilization in automation.
This commit replaces tc-vcs with the robustcheckout equivalent.
We replace the existing tc-vcs scope and cache with a new one.
Because Dustin and I are paranoid, we maintain separate caches per
SCM level - even though we could arguably share the same cache. Defense
in depth.
Robustcheckout (when used with --sharebase) pools storage for related
repos automatically. i.e. changesets from inbound and central will
be in the same store. This means you likely only have one copy of
each changeset per cache. This can result in significant space savings.
And, since there are fewer copies floating around, hg.mozilla.org
and various network appliances are working less too!
Since tc-vcs is no longer used, we stop it from being installed.
While we're here, we also change the images to execute as the
"worker" user. This happens automatically as a result of using
the "checkout-and-run" script.
MozReview-Commit-ID: EDeebuP7TkT
--HG--
extra : rebase_source : 2bec5dd9d6fe5565831bb35f195859aa12dd0bf2
extra : intermediate-source : 06481d97a485f6566554b087bc3880d76361e8ec
extra : source : d368700c93ef085325a081219d7aeb8512bc54a1
extra : histedit_source : c07505273fc8f10acf8e8d3ee01e327afd0aa63d
The script will be used as the main command in task YAML files.
It changes ownership of caches. Then switches to the "worker" user.
Then performs a Gecko checkout. Then executes whatever command was
requested via its arguments.
The script has been added to the shared recipes directory so it can
eventually be used by other Docker images. This means if we e.g. want
to add Git support, we only need to update one file in the tree.
MozReview-Commit-ID: Fuy1VrdSGYn
--HG--
extra : rebase_source : 407b2c584d56c95e9d9b23781539f2979a775893
extra : histedit_source : bd8b7fd541ed27da31082730ad3054b68b06544b
Like we do for the decision image, we install Mercurial 3.8.4 from deb
files hosted on tooltool. This provides more control and determinism
than installing via apt.
As part of this change, Mercurial is upgraded from whatever was hosted
in apt to 3.8.4.
Since the deb packages don't provide a global hgrc, we create one
ourselves. This is effectively copied from the decision image.
Most of the work is being done in a new, standalone
install-mercurial.sh script. This script is part of the
newly-established testing/docker/recipes directory. The intent of this
directory is to hold common files referenced by multiple images. Our
custom Dockerfile syntax to include files from outside the directory
with the Dockerfile is used to add these files to the build context.
MozReview-Commit-ID: K7gVm2Geihj
--HG--
extra : rebase_source : 6d1089ac34e43d399c7cf608d09eaaf405df00f7
extra : histedit_source : 656a4cea33ef913102b03238475461884c2749a0
Using our special Dockerfile syntax to include arbitrary files, we
include the previously vendored tooltool.py file in the image build
context and add it directly from there. No github.com communication
needed.
MozReview-Commit-ID: J42iXj87LEu
--HG--
extra : rebase_source : 90845e6793629b56998bf2fae2985913ee49c4eb
extra : histedit_source : 1fd5e64e40ae700efcf78b54e2a865b0594e0955
Visual aligning makes diffs harder to read. Use line continuations
to avoid this. Also make the package list alphabetical.
MozReview-Commit-ID: KqT4aqYyZfH
--HG--
extra : rebase_source : 08d2e4f61860bf6183ec3afaf598be158cd182be
extra : histedit_source : ff450a22617425214e90d42a6f1b530da8682847
This adds flake8 dependencies. Note that ubuntu 16.04 repos include pip > 8.0
which has peep merged into it, so there's no need to install peep separately.
I also ran into a locale issue which was causing a UnicodeDecodeError anytime
python tried to print a unicode character. The "locale-gen/dpkg-reconfigure locales"
fix the problem.
MozReview-Commit-ID: DQhYV6Emklw
--HG--
extra : transplant_source : %B8%EAh%CE%81%EF%95%1D%D5%B4%F5%AE%02V%20%7Em%83.%EB
This adds flake8 dependencies. Note that ubuntu 16.04 repos include pip > 8.0
which has peep merged into it, so there's no need to install peep separately.
I also ran into a locale issue which was causing a UnicodeDecodeError anytime
python tried to print a unicode character. The "locale-gen/dpkg-reconfigure locales"
fix the problem.
MozReview-Commit-ID: DQhYV6Emklw
--HG--
extra : rebase_source : ffefb07a17a43423142b3417628b6dd475d5c54a
The ES job uses an image based on the official node image. While this was convenient
for eslint, it is a bit less convenient for other things. I want to use this image
for all lint jobs, and switching the base to a generic ubuntu image seems a bit cleaner.
I chose 16.04 for no good reason other than it is the most recent, and we might as well.
Node v4.4.5 and taskcluster-vcs have been uploaded to tooltool.
MozReview-Commit-ID: GngYeH98zO1
--HG--
extra : rebase_source : 3c9b8264e159c601e1a3edad08d989272691751a
So a few changes here:
- node_modules is downloaded using tooltool so that we dont need to rely on external infrastructure.
- We have a npm-shrinkwrap.json file that version locks all of our node packages.
- eslint, eslint-plugin-mozilla etc. are now all installed locally.
In reality this means that we don't hit the network and we don't force users into installing global packages.
./mach eslint --setup has also been improved. We install packages locally and display the path of the user's eslint binary (useful for configuring editors).
eslint-plugin-mozilla has been moved from testing/eslint-plugin-mozilla to /testing/eslint/eslint-plugin-mozilla.
The node_modules directory for eslint and other plugins is located in testing/eslint/.
MozReview-Commit-ID: 4SFSxzka6BS
--HG--
rename : testing/eslint-plugin-mozilla/LICENSE => testing/eslint/eslint-plugin-mozilla/LICENSE
rename : testing/eslint-plugin-mozilla/docs/balanced-listeners.rst => testing/eslint/eslint-plugin-mozilla/docs/balanced-listeners.rst
rename : testing/eslint-plugin-mozilla/docs/import-browserjs-globals.rst => testing/eslint/eslint-plugin-mozilla/docs/import-browserjs-globals.rst
rename : testing/eslint-plugin-mozilla/docs/import-globals.rst => testing/eslint/eslint-plugin-mozilla/docs/import-globals.rst
rename : testing/eslint-plugin-mozilla/docs/import-headjs-globals.rst => testing/eslint/eslint-plugin-mozilla/docs/import-headjs-globals.rst
rename : testing/eslint-plugin-mozilla/docs/index.rst => testing/eslint/eslint-plugin-mozilla/docs/index.rst
rename : testing/eslint-plugin-mozilla/docs/mark-test-function-used.rst => testing/eslint/eslint-plugin-mozilla/docs/mark-test-function-used.rst
rename : testing/eslint-plugin-mozilla/docs/no-aArgs.rst => testing/eslint/eslint-plugin-mozilla/docs/no-aArgs.rst
rename : testing/eslint-plugin-mozilla/docs/no-cpows-in-tests.rst => testing/eslint/eslint-plugin-mozilla/docs/no-cpows-in-tests.rst
rename : testing/eslint-plugin-mozilla/docs/reject-importGlobalProperties.rst => testing/eslint/eslint-plugin-mozilla/docs/reject-importGlobalProperties.rst
rename : testing/eslint-plugin-mozilla/docs/var-only-at-top-level.rst => testing/eslint/eslint-plugin-mozilla/docs/var-only-at-top-level.rst
rename : testing/eslint-plugin-mozilla/lib/globals.js => testing/eslint/eslint-plugin-mozilla/lib/globals.js
rename : testing/eslint-plugin-mozilla/lib/helpers.js => testing/eslint/eslint-plugin-mozilla/lib/helpers.js
rename : testing/eslint-plugin-mozilla/lib/index.js => testing/eslint/eslint-plugin-mozilla/lib/index.js
rename : testing/eslint-plugin-mozilla/lib/processors/xbl-bindings.js => testing/eslint/eslint-plugin-mozilla/lib/processors/xbl-bindings.js
rename : testing/eslint-plugin-mozilla/lib/rules/.eslintrc => testing/eslint/eslint-plugin-mozilla/lib/rules/.eslintrc
rename : testing/eslint-plugin-mozilla/lib/rules/balanced-listeners.js => testing/eslint/eslint-plugin-mozilla/lib/rules/balanced-listeners.js
rename : testing/eslint-plugin-mozilla/lib/rules/import-browserjs-globals.js => testing/eslint/eslint-plugin-mozilla/lib/rules/import-browserjs-globals.js
rename : testing/eslint-plugin-mozilla/lib/rules/import-globals.js => testing/eslint/eslint-plugin-mozilla/lib/rules/import-globals.js
rename : testing/eslint-plugin-mozilla/lib/rules/import-headjs-globals.js => testing/eslint/eslint-plugin-mozilla/lib/rules/import-headjs-globals.js
rename : testing/eslint-plugin-mozilla/lib/rules/mark-test-function-used.js => testing/eslint/eslint-plugin-mozilla/lib/rules/mark-test-function-used.js
rename : testing/eslint-plugin-mozilla/lib/rules/no-aArgs.js => testing/eslint/eslint-plugin-mozilla/lib/rules/no-aArgs.js
rename : testing/eslint-plugin-mozilla/lib/rules/no-cpows-in-tests.js => testing/eslint/eslint-plugin-mozilla/lib/rules/no-cpows-in-tests.js
rename : testing/eslint-plugin-mozilla/lib/rules/reject-importGlobalProperties.js => testing/eslint/eslint-plugin-mozilla/lib/rules/reject-importGlobalProperties.js
rename : testing/eslint-plugin-mozilla/lib/rules/var-only-at-top-level.js => testing/eslint/eslint-plugin-mozilla/lib/rules/var-only-at-top-level.js
rename : testing/eslint-plugin-mozilla/moz.build => testing/eslint/eslint-plugin-mozilla/moz.build
rename : testing/eslint-plugin-mozilla/package.json => testing/eslint/eslint-plugin-mozilla/package.json
extra : rebase_source : cf689f6cc170b9a22018c981a768f545f952e019
Adds a new lint docker image for linting tools and adds an eslint-gecko task
that uses it to run eslint over the tree.
--HG--
extra : rebase_source : 6e3584ae9ec05ca1c45270f312d96cd026550e17
Gregory Szorc
Andrew Halberstadt
Carsten "Tomcat" Book
Michael Ratcliffe
J. Ryan Stinnett
Bob Silverberg
Dave Townsend