Changed flake8 installation strategy, try to update it using testings/lint/flake8_requirements.txt
moved flake8_requirements to tools/lint/flake8 directory
extracted common pip invocations to _run_pip function
minor fixes
fixed calls to check_process (concatenating list to tuple)
removed update_pip function and its error message variable
if flake8 installation reports and error print pip command output - replaced out with e.output in print function
MozReview-Commit-ID: Fmq22aL8dMq
--HG--
extra : rebase_source : 2e2f942c1ffab012795c632ceacf6debf0dc1407
When we initially implemented support for robustcheckout, we didn't
have the magic "%include" syntax in Dockerfiles. So we used tooltool
to download robustcheckout.py to the image.
Now that we have nice things, we can use the vendored robustcheckout.py
file.
As part of this, I realized we're inconsistently using /tmp, /setup
and /build for files used during image building. That should probably
be cleaned up. I'd rather not bloat scope for this bug, however.
MozReview-Commit-ID: D99Gcdw1DId
--HG--
extra : rebase_source : 3f361da3088988423b50786d990afba662d297d2
Before, we simply executed scripts inside Docker containers. This
frequently resulted in a wall of text with command output. It was
difficult to discern things like the time spent performing certain
actions.
Before, individual tasks had to drop permissions from the default
root user themselves. Dropping permissions isn't exactly a trivial
thing to do and a number of tasks didn't do it or did it wrong.
Before, we had a "checkout-gecko-and-run" script that kinda/sorta
did common activities for us. But it was written as a shell script
and doing advanced things was difficult.
This commit can be treated as a rewrite of "checkout-gecko-and-run"
as a Python script. But it also does a bit more. It prefixes output
with timestamps so we know how long operations took. It features more
robust argument parsing, so we can add new features more easily.
To prove the new wrapper script works, the lint image and all tasks
using it have been converted to use it.
MozReview-Commit-ID: 5d95u5Xebtq
--HG--
extra : rebase_source : 3a1d84782b01d7743e846bd0c04d7867813dd8a3
The reason tooltool.py was vendored in testing/docker/decision was
because locally built Docker images were using vanilla `docker build`
and didn't know about our special Dockerfile syntax to allow the
inclusion of images from outside the directory where the Dockerfile
was located.
Now that locally-built Docker images know of our special Dockerfile
syntax, we can include files from anywhere. So, move tooltool.py
to a shared directory, away from the decision image.
I didn't bump the version of the decision image because there are
a few more things I want to do to this image, such as have it use
the `checkout-gecko-and-run` script instead of its own script.
I think I'll do that in a separate bug, however.
MozReview-Commit-ID: 12ze9GJFMJq
--HG--
rename : testing/docker/decision/tooltool.py => testing/docker/recipes/tooltool.py
extra : rebase_source : cfc2fa5759a3ec708b6e1736e9a2da03f0d25428
The script will be used as the main command in task YAML files.
It changes ownership of caches. Then switches to the "worker" user.
Then performs a Gecko checkout. Then executes whatever command was
requested via its arguments.
The script has been added to the shared recipes directory so it can
eventually be used by other Docker images. This means if we e.g. want
to add Git support, we only need to update one file in the tree.
MozReview-Commit-ID: Fuy1VrdSGYn
--HG--
extra : rebase_source : 407b2c584d56c95e9d9b23781539f2979a775893
extra : histedit_source : bd8b7fd541ed27da31082730ad3054b68b06544b
Like we do for the decision image, we install Mercurial 3.8.4 from deb
files hosted on tooltool. This provides more control and determinism
than installing via apt.
As part of this change, Mercurial is upgraded from whatever was hosted
in apt to 3.8.4.
Since the deb packages don't provide a global hgrc, we create one
ourselves. This is effectively copied from the decision image.
Most of the work is being done in a new, standalone
install-mercurial.sh script. This script is part of the
newly-established testing/docker/recipes directory. The intent of this
directory is to hold common files referenced by multiple images. Our
custom Dockerfile syntax to include files from outside the directory
with the Dockerfile is used to add these files to the build context.
MozReview-Commit-ID: K7gVm2Geihj
--HG--
extra : rebase_source : 6d1089ac34e43d399c7cf608d09eaaf405df00f7
extra : histedit_source : 656a4cea33ef913102b03238475461884c2749a0
Using our special Dockerfile syntax to include arbitrary files, we
include the previously vendored tooltool.py file in the image build
context and add it directly from there. No github.com communication
needed.
MozReview-Commit-ID: J42iXj87LEu
--HG--
extra : rebase_source : 90845e6793629b56998bf2fae2985913ee49c4eb
extra : histedit_source : 1fd5e64e40ae700efcf78b54e2a865b0594e0955
The ES job uses an image based on the official node image. While this was convenient
for eslint, it is a bit less convenient for other things. I want to use this image
for all lint jobs, and switching the base to a generic ubuntu image seems a bit cleaner.
I chose 16.04 for no good reason other than it is the most recent, and we might as well.
Node v4.4.5 and taskcluster-vcs have been uploaded to tooltool.
MozReview-Commit-ID: GngYeH98zO1
--HG--
extra : rebase_source : 3c9b8264e159c601e1a3edad08d989272691751a
So a few changes here:
- node_modules is downloaded using tooltool so that we dont need to rely on external infrastructure.
- We have a npm-shrinkwrap.json file that version locks all of our node packages.
- eslint, eslint-plugin-mozilla etc. are now all installed locally.
In reality this means that we don't hit the network and we don't force users into installing global packages.
./mach eslint --setup has also been improved. We install packages locally and display the path of the user's eslint binary (useful for configuring editors).
eslint-plugin-mozilla has been moved from testing/eslint-plugin-mozilla to /testing/eslint/eslint-plugin-mozilla.
The node_modules directory for eslint and other plugins is located in testing/eslint/.
MozReview-Commit-ID: 4SFSxzka6BS
--HG--
rename : testing/eslint-plugin-mozilla/LICENSE => testing/eslint/eslint-plugin-mozilla/LICENSE
rename : testing/eslint-plugin-mozilla/docs/balanced-listeners.rst => testing/eslint/eslint-plugin-mozilla/docs/balanced-listeners.rst
rename : testing/eslint-plugin-mozilla/docs/import-browserjs-globals.rst => testing/eslint/eslint-plugin-mozilla/docs/import-browserjs-globals.rst
rename : testing/eslint-plugin-mozilla/docs/import-globals.rst => testing/eslint/eslint-plugin-mozilla/docs/import-globals.rst
rename : testing/eslint-plugin-mozilla/docs/import-headjs-globals.rst => testing/eslint/eslint-plugin-mozilla/docs/import-headjs-globals.rst
rename : testing/eslint-plugin-mozilla/docs/index.rst => testing/eslint/eslint-plugin-mozilla/docs/index.rst
rename : testing/eslint-plugin-mozilla/docs/mark-test-function-used.rst => testing/eslint/eslint-plugin-mozilla/docs/mark-test-function-used.rst
rename : testing/eslint-plugin-mozilla/docs/no-aArgs.rst => testing/eslint/eslint-plugin-mozilla/docs/no-aArgs.rst
rename : testing/eslint-plugin-mozilla/docs/no-cpows-in-tests.rst => testing/eslint/eslint-plugin-mozilla/docs/no-cpows-in-tests.rst
rename : testing/eslint-plugin-mozilla/docs/reject-importGlobalProperties.rst => testing/eslint/eslint-plugin-mozilla/docs/reject-importGlobalProperties.rst
rename : testing/eslint-plugin-mozilla/docs/var-only-at-top-level.rst => testing/eslint/eslint-plugin-mozilla/docs/var-only-at-top-level.rst
rename : testing/eslint-plugin-mozilla/lib/globals.js => testing/eslint/eslint-plugin-mozilla/lib/globals.js
rename : testing/eslint-plugin-mozilla/lib/helpers.js => testing/eslint/eslint-plugin-mozilla/lib/helpers.js
rename : testing/eslint-plugin-mozilla/lib/index.js => testing/eslint/eslint-plugin-mozilla/lib/index.js
rename : testing/eslint-plugin-mozilla/lib/processors/xbl-bindings.js => testing/eslint/eslint-plugin-mozilla/lib/processors/xbl-bindings.js
rename : testing/eslint-plugin-mozilla/lib/rules/.eslintrc => testing/eslint/eslint-plugin-mozilla/lib/rules/.eslintrc
rename : testing/eslint-plugin-mozilla/lib/rules/balanced-listeners.js => testing/eslint/eslint-plugin-mozilla/lib/rules/balanced-listeners.js
rename : testing/eslint-plugin-mozilla/lib/rules/import-browserjs-globals.js => testing/eslint/eslint-plugin-mozilla/lib/rules/import-browserjs-globals.js
rename : testing/eslint-plugin-mozilla/lib/rules/import-globals.js => testing/eslint/eslint-plugin-mozilla/lib/rules/import-globals.js
rename : testing/eslint-plugin-mozilla/lib/rules/import-headjs-globals.js => testing/eslint/eslint-plugin-mozilla/lib/rules/import-headjs-globals.js
rename : testing/eslint-plugin-mozilla/lib/rules/mark-test-function-used.js => testing/eslint/eslint-plugin-mozilla/lib/rules/mark-test-function-used.js
rename : testing/eslint-plugin-mozilla/lib/rules/no-aArgs.js => testing/eslint/eslint-plugin-mozilla/lib/rules/no-aArgs.js
rename : testing/eslint-plugin-mozilla/lib/rules/no-cpows-in-tests.js => testing/eslint/eslint-plugin-mozilla/lib/rules/no-cpows-in-tests.js
rename : testing/eslint-plugin-mozilla/lib/rules/reject-importGlobalProperties.js => testing/eslint/eslint-plugin-mozilla/lib/rules/reject-importGlobalProperties.js
rename : testing/eslint-plugin-mozilla/lib/rules/var-only-at-top-level.js => testing/eslint/eslint-plugin-mozilla/lib/rules/var-only-at-top-level.js
rename : testing/eslint-plugin-mozilla/moz.build => testing/eslint/eslint-plugin-mozilla/moz.build
rename : testing/eslint-plugin-mozilla/package.json => testing/eslint/eslint-plugin-mozilla/package.json
extra : rebase_source : cf689f6cc170b9a22018c981a768f545f952e019
Adds a new lint docker image for linting tools and adds an eslint-gecko task
that uses it to run eslint over the tree.
--HG--
extra : rebase_source : 6e3584ae9ec05ca1c45270f312d96cd026550e17