This also turns the tier 2 job B(n)g into tier 1, since moz.build is
still tier 1. It also pushes a lot of GeckoView related tasks into
the main builds, since they should run as part of Gradle builds.
This also removes unused tooltool manifests; the jobs that used these
manifests use only toolchain tasks now.
MozReview-Commit-ID: 2GmnJ7joCTT
--HG--
extra : rebase_source : 75cd2dfb51e0e1b510f5e618c2dc881cf5f22bf2
extra : source : 6b95b09d6afbb83ba89c47b237dfce6e15587bbe
A lot of effort has been spent optimizing VCS operations for peak
performance. But not utilizing caches or volumes for the VCS store
or checkouts can undermine that work.
Let's print a warning when VCS is configured sub-optimally.
I'm pretty sure we still have some rogue tasks not using caches
or volumes. We can convert this to a fatal error once those are
fixed.
MozReview-Commit-ID: C6CT1zViy75
--HG--
extra : rebase_source : 91760250bed263c789b95d16cc0542a53ca2bfbf
This seems like a reasonable thing to enforce.
MozReview-Commit-ID: 3BZQSkwRYeN
--HG--
extra : rebase_source : 8dae62edb35202da0f0e90ddec3eacb212ada371
We introduce a per-cache .cachelog file containing important events
in the cache's history such as creation, requirements adjusting,
and utilization. If cache validation fails, we print the cache log.
If a previous task was responsible for getting the cache in a bad
state, its TASK_ID should be printed, allowing us to more easily
identify mis-configured tasks.
MozReview-Commit-ID: BJun5Hi5w0s
--HG--
extra : rebase_source : f4758741ee294a0de53882b6891b473c01463e28
AFAICT, the last use of the build.sh script was removed in bug 1309593,
and checkout-sources.sh is only used from that script.
--HG--
extra : rebase_source : f89d9071af0e49a168e1eb7c38e291c29ed7119f
CentOS 6 is pinned to glibc 2.12, but newer Android build-tools (like
aapt) require glibc 2.14. It's not possible to safely upgrade CentOS
6 distributions to glibc 2.14.
CentOS 7 is pinned to glibc 2.17, which is new enough for newer
Android build-tools. However, I had great difficulty bringing forward
our existing centos:6 Docker image to centos:7. In particular,
installing recent enough Mercurial, git, Python, and pip versions was
difficult enough that I elected to not pursue this approach.
Instead, I've elected to follow glandium's suggestion from
https://bugzilla.mozilla.org/show_bug.cgi?id=1370119#c5: base on
Debian with snapshots.debian.org for reproducibility.
The most significant changes here:
- using Debian's snapshots repository
- using Python and related tools provided by Debian and baked into the
build image
- using the JDK and JRE provided by Debian and baked into the build
image, rather than versions from tooltool (or eventually a toolchain
build)
Moving the builds over to use this image will follow in the patches
ahead.
CLOSED TREE
--HG--
extra : amend_source : 84120d6bacb5d72a9fbe41e4c3b405d63825da7c
extra : histedit_source : 8320c2193761b745f10850055ee74a3c9ac73615%2Cfbc2a28d8c5004a53305ef858ca5aea4245691e0
CentOS 6 is pinned to glibc 2.12, but newer Android build-tools (like
aapt) require glibc 2.14. It's not possible to safely upgrade CentOS
6 distributions to glibc 2.14.
CentOS 7 is pinned to glibc 2.17, which is new enough for newer
Android build-tools. However, I had great difficulty bringing forward
our existing centos:6 Docker image to centos:7. In particular,
installing recent enough Mercurial, git, Python, and pip versions was
difficult enough that I elected to not pursue this approach.
Instead, I've elected to follow glandium's suggestion from
https://bugzilla.mozilla.org/show_bug.cgi?id=1370119#c5: base on
Debian with snapshots.debian.org for reproducibility.
The most significant changes here:
- using Debian's snapshots repository
- using Python and related tools provided by Debian and baked into the
build image
- using the JDK and JRE provided by Debian and baked into the build
image, rather than versions from tooltool (or eventually a toolchain
build)
Moving the builds over to use this image will follow in the patches
ahead.
The name `android-gradle-build` is an accident of history; let's rename it
before we attempt major surgery on it.
--HG--
rename : taskcluster/docker/android-gradle-build/Dockerfile => taskcluster/docker/android-build/Dockerfile
rename : taskcluster/docker/android-gradle-build/README.md => taskcluster/docker/android-build/README.md
rename : taskcluster/docker/android-gradle-build/REGISTRY => taskcluster/docker/android-build/REGISTRY
rename : taskcluster/docker/android-gradle-build/VERSION => taskcluster/docker/android-build/VERSION
rename : taskcluster/docker/android-gradle-build/buildprops.json => taskcluster/docker/android-build/buildprops.json
rename : taskcluster/docker/android-gradle-build/dot-config/pip/pip.conf => taskcluster/docker/android-build/dot-config/pip/pip.conf
rename : taskcluster/docker/android-gradle-build/oauth.txt => taskcluster/docker/android-build/oauth.txt
This includes adding TASKCLUSTER_VOLUMES to docker image builds directly. The
env variable is not added as part of the task transform because `run-task` is
not in payload.command. In fact, build-image.sh calls run-task after doing
some other housekeeping.
Ideally image builds would be turned into jobs and all of this would occur
automatically, but that turns out to be quite a bit too complex for this
incidental fix -- perhaps best solved in another bug.
MozReview-Commit-ID: FYHvafJras7
--HG--
extra : rebase_source : 4e3b9ae9900727e7932c13ced34b3f8596d755d9
This will allow us keep python related linting files in the same place.
MozReview-Commit-ID: ABtq9dnPo9T
--HG--
rename : tools/lint/flake8_/__init__.py => tools/lint/python/__init__.py
rename : tools/lint/flake8_/__init__.py => tools/lint/python/flake8.py
rename : tools/lint/flake8_/flake8_requirements.txt => tools/lint/python/flake8_requirements.txt
extra : rebase_source : 2568bc0bf8f4adbf8e0be73a54d5da068a8d81b0
This includes adding TASKCLUSTER_VOLUMES to docker image builds directly. The
env variable is not added as part of the task transform because `run-task` is
not in payload.command. In fact, build-image.sh calls run-task after doing
some other housekeeping.
Ideally image builds would be turned into jobs and all of this would occur
automatically, but that turns out to be quite a bit too complex for this
incidental fix -- perhaps best solved in another bug.
MozReview-Commit-ID: FYHvafJras7
--HG--
extra : rebase_source : 4e3b9ae9900727e7932c13ced34b3f8596d755d9
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: 554IPMRWgzK
--HG--
extra : rebase_source : 00827d3f6bd705419bc801eb05b543af1ddc274f
The updated Docker image contains robustcheckout and run-task support
for sparse checkouts, which are obvious prerequisites.
We change the cache name so sparse and non-sparse checkouts don't
use the same working directory. If we didn't do this, tasks running
from images with old Mercurial clients or without a sparse aware
robustcheckout would fail.
The effect of using a sparse checkout is that we reduce the number
of files in the checkout from ~234,000 to ~3,600. This reduces time
for a fresh checkout from several dozen seconds to under 2s.
MozReview-Commit-ID: IJz794g8ZKH
--HG--
extra : source : 9923fffd4f64a1aa9d762e6027e0e2424a19c49c
The updated Docker image contains robustcheckout and run-task support
for sparse checkouts, which are obvious prerequisites.
We change the cache name so sparse and non-sparse checkouts don't
use the same working directory. If we didn't do this, tasks running
from images with old Mercurial clients or without a sparse aware
robustcheckout would fail.
The effect of using a sparse checkout is that we reduce the number
of files in the checkout from ~234,000 to ~3,600. This reduces time
for a fresh checkout from several dozen seconds to under 2s.
MozReview-Commit-ID: IJz794g8ZKH
--HG--
extra : rebase_source : d262c8314381a136cf5cdc5c33669c8c61818d1d
Rail Aliiev
Nick Alexander
Ben Hearsum
Gregory Szorc
Mike Hommey
Tom Ritter
Johan Lorenzo
Nick Thomas
Wes Kocher
Simon Fraser
Sebastian Hengst
Jonas Finnemann Jensen
Dustin J. Mitchell
Phil Ringnalda
Andrew Halberstadt
Wander Lairson Costa