This adds a download of allthethings.json to the Decision Task (in optimize phase).
If there is a network error, in order to avoid a broken decision task, we treat all buildbot-bridge jobs as valid.
MozReview-Commit-ID: GpKVV5pUwGL
--HG--
extra : rebase_source : 39e853f98dcfafe252504f724cdf84b95b0eece8
Strictly speaking we don't need all files in the directories listed
in the profile. But the checkout is still small enough and it is far
less effort than cherry-picking every file needed by every toolchain
task.
This brings the checkout down to ~3700 files, which only takes 1-2s.
MozReview-Commit-ID: 2BpKdZ2Pvx9
--HG--
extra : rebase_source : e9589b29f7e1b60ac7ee4e8050689dc3d5a8f418
extra : source : 2c0d6e566cc719823515ec2403e1d6b2ace955aa
This continues to use a file-based reader when run locally.
MozReview-Commit-ID: CJuYKDj2E3n
--HG--
extra : rebase_source : ceb56f3eb5b56fca90f19736ab710696fde86bd1
This means that a push to try affecting only Android will only run android builds
and tests, for example.
MozReview-Commit-ID: HVUvIg0EUZn
--HG--
extra : rebase_source : 8b080b7648cea5f82cd03c9a7950667277b75118
extra : source : b41cd667697e13c989659b16bf649090a3908ecd
This adds some new optimization strategies. For tests, we use Either(SETA,
SkipUnlessSchedules), thereby giving both mechanisms a chance to skip tasks. On
try, SETA is omitted.
MozReview-Commit-ID: GL4tlwyeBa6
--HG--
extra : rebase_source : f208e4960cf76a9dfe77634b87f0058f676e9fa9
extra : source : 046d705929f7a41e977eec19c8503afccdec7592
The previous patch was ineffective, because the noted transform over-wrote
the yml configuration.
MozReview-Commit-ID: ICENT0DGzxy
--HG--
extra : source : e024769dac36e22935f1e64a328bb15334d6bdef
This was preventing us from running ./mach try fuzzy from a subdirectory of the
topsrcdir. This also fixes taskgraph/utils/verify.py to find the docs directory
based off of topsrcdir instead of cwd. This was needed as cwd was being set at
import time.
MozReview-Commit-ID: CgQqD6bQ5q4
--HG--
extra : rebase_source : 4d863d442b0e715b3fe386c43b4077054c10dc70
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.
The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.
MozReview-Commit-ID: AXJEGLh6pEV
--HG--
extra : rebase_source : 25e9966696d78d899783d9f38533d5ae66f9ccb9
extra : source : b53ff084c2d7968a1d9864d1343f2d9381fb652b
In preparation for much more thorough optimization of task-graphs, this
makes a few changes:
* optimization is split into thre phases, with task removal in one phase
(following dependency links) and task replacement in the next (in the
reverse order).
* optimization uses class instances instead of functions for optimizations;
this allows different functions for different phases, and also leaves open
the possibility of composing optimizations.
* the replacement phase can also support removal; this is when utility tasks
like symbol uploads can be optimized away iff their parent task is
optimized.
MozReview-Commit-ID: C5QznNpwqXn
--HG--
extra : rebase_source : cf1654036041a64398a2cd38e35e8de4f3596ff9
extra : source : c25af2c111a5be4e3381d0b002641691d15fe4e8
It is not at *all* clear how multiple optimizations for a single task should
interact. No simple logical operation is right in all cases, and in fact in
most imaginable cases the desired behavior turns out to be independent of all
but one of the optimizations. For example, given both `seta` and
`skip-unless-files-changed` optimizations, if SETA says to skip a test, it is
low value and should be skipped regardless of what files have changed. But if
SETA says to run a test, then it has likely been skipped in previous pushes, so
it should be run regardless of what has changed in this push.
This also adds a bit more output about optimization, that may be useful for
anyone wondering why a particular job didn't run.
MozReview-Commit-ID: 3OsvRnWjai4
--HG--
extra : rebase_source : ba0aa536e8c474b36c63d1447c83ed9885f1e3e6
extra : source : a3b7bdfdb116300daa3f49e0dfc96177e1369440
Strictly speaking we don't need all files in the directories listed
in the profile. But the checkout is still small enough and it is far
less effort than cherry-picking every file needed by every toolchain
task.
This brings the checkout down to ~3700 files, which only takes 1-2s.
MozReview-Commit-ID: 2BpKdZ2Pvx9
--HG--
extra : rebase_source : 916c6722c6d495cdccea0076c673d75e4ee48f28
extra : source : 2c0d6e566cc719823515ec2403e1d6b2ace955aa
Now that we have a Docker image with newer library versions on it, we
can move our builds over. The new images differ from the old
CentOS-based images in two important ways, though:
1) The system compilers in the new image are new enough to be used as
host compilers; additionally, our CentOS-built GCC compilers will not
work. We need to change the Android mozconfigs to reflect that. We
also need to change the Android tasks to not depend on the GCC
toolchain builds.
2) In a similar fashion, we can use the system JDK; we no longer need to
use the JDK from the Android NDK, which we had packaged up via the
Android dependencies task.
Both of these changes come with caveats: our l10n repack jobs continue
to run on the CentOS-based images; l10n repacks have not been completely
converted to Taskcluster. So we need to:
1) Retain the use of our custom GCC toolchain for HOST_CC/HOST_CXX on
the CentOS-based images.
2) Retain the JDK packages in the tooltool manifests, and referencing
them when we build on the CentOS-based images.
CentOS 6 is pinned to glibc 2.12, but newer Android build-tools (like
aapt) require glibc 2.14. It's not possible to safely upgrade CentOS
6 distributions to glibc 2.14.
CentOS 7 is pinned to glibc 2.17, which is new enough for newer
Android build-tools. However, I had great difficulty bringing forward
our existing centos:6 Docker image to centos:7. In particular,
installing recent enough Mercurial, git, Python, and pip versions was
difficult enough that I elected to not pursue this approach.
Instead, I've elected to follow glandium's suggestion from
https://bugzilla.mozilla.org/show_bug.cgi?id=1370119#c5: base on
Debian with snapshots.debian.org for reproducibility.
The most significant changes here:
- using Debian's snapshots repository
- using Python and related tools provided by Debian and baked into the
build image
- using the JDK and JRE provided by Debian and baked into the build
image, rather than versions from tooltool (or eventually a toolchain
build)
Moving the builds over to use this image will follow in the patches
ahead.
I also snuck in some last-minute assertions and minor fixes into this patch:
- don't stop reporting for a callee if we've seen it already (or rather, make the reachable set local to a root rather than global to all roots). This slows down runs with hundreds of hazards, but results in every problematic root being reported, for a more accurate count.
- annotate away some thread assertions
- special-case annotation for bug 1400435 since it's a whole family of hazards
--HG--
extra : rebase_source : ac7335d45e3e0772d34cb42cc6a3f628564fd3d1
CLOSED TREE
--HG--
extra : amend_source : 84120d6bacb5d72a9fbe41e4c3b405d63825da7c
extra : histedit_source : 8320c2193761b745f10850055ee74a3c9ac73615%2Cfbc2a28d8c5004a53305ef858ca5aea4245691e0
Now that we have a Docker image with newer library versions on it, we
can move our builds over. The new images differ from the old
CentOS-based images in two important ways, though:
1) The system compilers in the new image are new enough to be used as
host compilers; additionally, our CentOS-built GCC compilers will not
work. We need to change the Android mozconfigs to reflect that. We
also need to change the Android tasks to not depend on the GCC
toolchain builds.
2) In a similar fashion, we can use the system JDK; we no longer need to
use the JDK from the Android NDK, which we had packaged up via the
Android dependencies task.
Both of these changes come with caveats: our l10n repack jobs continue
to run on the CentOS-based images; l10n repacks have not been completely
converted to Taskcluster. So we need to:
1) Retain the use of our custom GCC toolchain for HOST_CC/HOST_CXX on
the CentOS-based images.
2) Retain the JDK packages in the tooltool manifests, and referencing
them when we build on the CentOS-based images.
CentOS 6 is pinned to glibc 2.12, but newer Android build-tools (like
aapt) require glibc 2.14. It's not possible to safely upgrade CentOS
6 distributions to glibc 2.14.
CentOS 7 is pinned to glibc 2.17, which is new enough for newer
Android build-tools. However, I had great difficulty bringing forward
our existing centos:6 Docker image to centos:7. In particular,
installing recent enough Mercurial, git, Python, and pip versions was
difficult enough that I elected to not pursue this approach.
Instead, I've elected to follow glandium's suggestion from
https://bugzilla.mozilla.org/show_bug.cgi?id=1370119#c5: base on
Debian with snapshots.debian.org for reproducibility.
The most significant changes here:
- using Debian's snapshots repository
- using Python and related tools provided by Debian and baked into the
build image
- using the JDK and JRE provided by Debian and baked into the build
image, rather than versions from tooltool (or eventually a toolchain
build)
Moving the builds over to use this image will follow in the patches
ahead.
The name `android-gradle-build` is an accident of history; let's rename it
before we attempt major surgery on it.
--HG--
rename : taskcluster/docker/android-gradle-build/Dockerfile => taskcluster/docker/android-build/Dockerfile
rename : taskcluster/docker/android-gradle-build/README.md => taskcluster/docker/android-build/README.md
rename : taskcluster/docker/android-gradle-build/REGISTRY => taskcluster/docker/android-build/REGISTRY
rename : taskcluster/docker/android-gradle-build/VERSION => taskcluster/docker/android-build/VERSION
rename : taskcluster/docker/android-gradle-build/buildprops.json => taskcluster/docker/android-build/buildprops.json
rename : taskcluster/docker/android-gradle-build/dot-config/pip/pip.conf => taskcluster/docker/android-build/dot-config/pip/pip.conf
rename : taskcluster/docker/android-gradle-build/oauth.txt => taskcluster/docker/android-build/oauth.txt
Wes Kocher
Sebastian Hengst
Brian Stack
Justin Wood
Gregory Szorc
Dustin J. Mitchell
Johan Lorenzo
Geoff Brown
Rob Wood
Andrew Halberstadt
Simon Fraser
Nathan Froyd
Nick Alexander
Rail Aliiev
Joel Maher
Steve Fink
Nick Thomas
Ryan VanderMeulen