Copy the repack_rust.py from the rust-build docker container
so it can be used more generally by other taskcluster jobs.
Add --host, --target, and --suffix switches, allowing control
of the packaged toolchain and standard library builds from
the command line.
This drops the previous default behaviour of packaging all
supported platforms and targets.
Add a hard-coded copy of the Rust release signing key to
the script and add it to the running user's gpg config
so we can validate downloaded artifacts from the project
in automation.
Remove the keybase artifact validation since it requires
out-of-project network services and doesn't provide much
advantage in automation.
Calculate the SHA-2 checksum during download and remove
the dependency on shasum/sha256sum command-line tools.
Use more python for filesystem an process interaction
in general.
Create a generic rustc.tar.* package to correctly match
the unversioned unpack dirctory name.
Add support for copying the package to an output directory
if the UPLOAD_DIR environment variable is set. This lets
us hook up the script to taskcluster toolchain jobs without
an external wrapper.
MozReview-Commit-ID: 68LmY3QVU8V
--HG--
extra : rebase_source : f6329056d518ad2cd25faa5c71b22130cbc65c8f
CentOS 6 is pinned to glibc 2.12, but newer Android build-tools (like
aapt) require glibc 2.14. It's not possible to safely upgrade CentOS
6 distributions to glibc 2.14.
CentOS 7 is pinned to glibc 2.17, which is new enough for newer
Android build-tools. However, I had great difficulty bringing forward
our existing centos:6 Docker image to centos:7. In particular,
installing recent enough Mercurial, git, Python, and pip versions was
difficult enough that I elected to not pursue this approach.
Instead, I've elected to follow glandium's suggestion from
https://bugzilla.mozilla.org/show_bug.cgi?id=1370119#c5: base on
Debian with snapshots.debian.org for reproducibility.
The most significant changes here:
- using Debian's snapshots repository
- using Python and related tools provided by Debian and baked into the
build image
- using the JDK and JRE provided by Debian and baked into the build
image, rather than versions from tooltool (or eventually a toolchain
build)
Moving the builds over to use this image will follow in the patches
ahead.
I also snuck in some last-minute assertions and minor fixes into this patch:
- don't stop reporting for a callee if we've seen it already (or rather, make the reachable set local to a root rather than global to all roots). This slows down runs with hundreds of hazards, but results in every problematic root being reported, for a more accurate count.
- annotate away some thread assertions
- special-case annotation for bug 1400435 since it's a whole family of hazards
--HG--
extra : rebase_source : ac7335d45e3e0772d34cb42cc6a3f628564fd3d1
CLOSED TREE
--HG--
extra : amend_source : 84120d6bacb5d72a9fbe41e4c3b405d63825da7c
extra : histedit_source : 8320c2193761b745f10850055ee74a3c9ac73615%2Cfbc2a28d8c5004a53305ef858ca5aea4245691e0
CentOS 6 is pinned to glibc 2.12, but newer Android build-tools (like
aapt) require glibc 2.14. It's not possible to safely upgrade CentOS
6 distributions to glibc 2.14.
CentOS 7 is pinned to glibc 2.17, which is new enough for newer
Android build-tools. However, I had great difficulty bringing forward
our existing centos:6 Docker image to centos:7. In particular,
installing recent enough Mercurial, git, Python, and pip versions was
difficult enough that I elected to not pursue this approach.
Instead, I've elected to follow glandium's suggestion from
https://bugzilla.mozilla.org/show_bug.cgi?id=1370119#c5: base on
Debian with snapshots.debian.org for reproducibility.
The most significant changes here:
- using Debian's snapshots repository
- using Python and related tools provided by Debian and baked into the
build image
- using the JDK and JRE provided by Debian and baked into the build
image, rather than versions from tooltool (or eventually a toolchain
build)
Moving the builds over to use this image will follow in the patches
ahead.
Our current sccache build links in openssl's libraries dynamically. The
sonames of the dynamic libraries linked in are specific to the
CentOS/Fedora-ish systems that we build on; attempting to run the
generated sccache binaries on different systems (e.g. Debian-ish) will
result in failure. All of our current automation images are
CentOS-based, but for various reasons, Debian-based images may be used
in the future, and it would be great to have an sccache binary to run on
such systems as well. (It might also be interesting to distribute the
sccache binary we use to local developers as well, but that's a bit
further off.)
Therefore, this patch alters the sccache build on Linux to use static
linking for openssl. We cannot use the system openssl we build on
because the system openssl links to libkrb5, and the distribution we use
for the system images does not provide static libraries of libkrb5.
Building openssl ourself enables us to eliminate the libkrb5 dependency.
An sccache binary from builds with this patch depends on the following
libraries:
froydnj@hawkeye:~$ ldd sccache2/sccache
linux-vdso.so.1 => (0x00007ffe02b39000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff0e7403000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff0e71fb000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff0e6fdd000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff0e6dc6000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff0e69fc000)
/lib64/ld-linux-x86-64.so.2 (0x0000557c8540b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff0e66f2000)
which are standard on any Linux distribution.
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: 554IPMRWgzK
--HG--
extra : rebase_source : 00827d3f6bd705419bc801eb05b543af1ddc274f
We're about to ban files in Docker volumes so they behave almost
identically to caches (which start empty).
We move the install of nexus.xml from Docker image time to
task time. This also means that changes to nexus.xml don't result
in having to rebuild the Docker image.
MozReview-Commit-ID: JIjeJN4mt2
--HG--
rename : taskcluster/docker/android-gradle-build/nexus.xml => taskcluster/scripts/builder/build-android-dependencies/nexus.xml
extra : rebase_source : 53848f06820bda7979b2ae15456e07f8aed2363d
This picks up a fix we need to update the OS X SDK we build with.
MozReview-Commit-ID: 8dvq4JV1o7q
--HG--
extra : rebase_source : a07f13992f30a29ede29a2167e7f1da8d533fd09
AFAICT there are no more in-tree references to this image. That
should mean we can nuke it. So do that.
MozReview-Commit-ID: 9LUGjt46ZCi
--HG--
extra : rebase_source : caa9e8f3e355710542794efb7f6f92c2ef43ef0a
The old process ran "before" and "after" steps as root. The
mozharness script doesn't run as root, which required some small
changes to not run Sonatype Nexus as root. Everything else is a
straight-forward move of the scripts out of the `android-gradle-build`
image and into `taskcluster/scripts`.
MozReview-Commit-ID: CqnNI33OKmb
--HG--
rename : taskcluster/docker/android-gradle-build/bin/after.sh => taskcluster/scripts/builder/build-android-dependencies/after.sh
rename : taskcluster/docker/android-gradle-build/bin/before.sh => taskcluster/scripts/builder/build-android-dependencies/before.sh
rename : taskcluster/docker/android-gradle-build/bin/repackage-jdk-centos.sh => taskcluster/scripts/builder/build-android-dependencies/repackage-jdk-centos.sh
extra : rebase_source : f94e6b9b780f96038c60d3825039a0f94add0404
While this looks like going backwards, it is desirable to build clang
against GCC 4.8, such that it contains its libgcc. This, in turn, will
solve problems using clang 3.9 with static-analysis builds (details in
bug 1356926). Another way to fix those problems would be to build clang
3.8 but that too would require GCC 4.8. Upgrading those builds to clang
3.9 will also allow to enable stylo on them.
It becomes a library of some sort, so that multiple scripts can benefit
from it to build different versions of GCC.
The GPG key associated with GCC is also refreshed from keys.gnupg.net,
adding a new subkey, used to sign newer versions of GCC (and
postprocessed with pgpstrip to make it smaller).
We're soon going to build multiple versions of clang and gcc for linux,
and we need to differentiate them. Furthermore, there is a need for the
base-toolchains builds to use a fixed version of clang and gcc. So
rename the clang and gcc toolchain jobs to include their version, add
aliases to satisfy all existing jobs, and adjust the base-toolchains
jobs to use the explicit version.
--HG--
rename : build/build-clang/clang-linux64.json => build/build-clang/clang-3.9-linux64.json
rename : taskcluster/scripts/misc/build-gcc-linux.sh => taskcluster/scripts/misc/build-gcc-4.9-linux.sh
Allow an extra heap write hazard introduced by enabling stylo
in default builds until it can be addressed. See bug 1384625.
MozReview-Commit-ID: 2N3z6FVHa0G
--HG--
extra : rebase_source : d4c9c18c34ab829b59b9e16ec3cd8e736c418fd3
With the support added in bug 1382564, toolchains can be downloaded
without a tooltool manifest at all, and it's desirable to get rid of
tooltool manifests on jobs that don't need one.
--HG--
extra : rebase_source : 340d935034133f4f6d8aec3d15154736092859fa
The MinGit tooltool package used for Windows builds comes straight from
https://github.com/git-for-windows/git/releases/
This builds the version currently used on automation.
--HG--
extra : rebase_source : dbc2a36b07611e673d6661032ad53123a688d422
This affects the location of the artifacts directory, so adjust the
scripts and artifact definitions as a consequence.
--HG--
extra : rebase_source : 008b8cf33957eabfcacee61de8a260b5df146ab4
Bug 1374940 adds a MOZ_TOOLCHAINS environment variable with a list of
path@task-id strings, where task-id is corresponding to the (possibly
optimized) toolchain job, and path corresponding to the
toolchain-artifact defined for that toolchain job.
We want to use that to pull artifacts instead of tooltool packages.
--HG--
extra : rebase_source : 277daa2c83d6d197975cb4ef36ee131176afa992
This is a quick work around for not blocking the progress of Bug 1380133.
We should definely investigate the real root cause sooner than later.
MozReview-Commit-ID: 8X1FH6f2GyN
--HG--
extra : rebase_source : 9edaec6a50f7251e3c05d9fbc252e1385d82a1bb
This is a quick work around for not blocking the progress of Bug 1380133.
We should definely investigate the real root cause sooner than later.
MozReview-Commit-ID: 8X1FH6f2GyN
We're about to make MOZ_AUTOMATION more strict about things like having
a source checkout.
The whole point of build-sm-package.sh is to verify that SpiderMonkey
can be built outside of Mozilla's source repo and automation from a
standalone package. Since the presence of MOZ_AUTOMATION can influence
so much behavior in the build system, unset it so that the job
tests a !Mozilla environment more accurately.
MozReview-Commit-ID: EMfyLKfY0uU
--HG--
extra : rebase_source : 3632a9abf9fac3f916ed9043f30d4b6aa4abb390
We're about to make MOZ_AUTOMATION more strict about things like having
a source checkout.
The whole point of build-sm-package.sh is to verify that SpiderMonkey
can be built outside of Mozilla's source repo and automation from a
standalone package. Since the presence of MOZ_AUTOMATION can influence
so much behavior in the build system, unset it so that the job
tests a !Mozilla environment more accurately.
MozReview-Commit-ID: EMfyLKfY0uU
--HG--
extra : rebase_source : 3632a9abf9fac3f916ed9043f30d4b6aa4abb390
The scripts that use tooltool-download.sh don't run regularly, but when
they do, they might hit some download problems (the relengapi proxy
tends to be rather unreliable for some reason), and in that case, it
would be better to retry a few times, like other job types, rather than
fail directly.
--HG--
extra : rebase_source : d85797f8eebff9be5b8bfc45fc14dfaf8d5a59f3
All the current users of tooltool-download.sh set $WORKSPACE. This will
allow to reuse the script on different types of workers, that don't have
$WORKSPACE set to $HOME/workspace, but still have the source in
$WORKSPACE/build/src.
--HG--
extra : rebase_source : c91542da70109e5708a009542a0f588e30b541a9
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: zehcGJrUQX
--HG--
extra : source : feedcde68c2a54da210f03eb287ab5c862fc982b
extra : intermediate-source : 485d1af7805ad9fa0e701c3571fc1291fbfc6850
The toolchain build scripts are currently defining the tooltool
manifest they use on their own. We move the definitions to the
taskcluster job definitions to normalize on everything using that.
--HG--
extra : rebase_source : cbab2e32d78d711fcb595763d0c2600c7c0c423e
Tom Ritter
Ralph Giles
Nick Alexander
Steve Fink
Gregory Szorc
Stephen A Pohl
Nick Fitzgerald
Nathan Froyd
Chris Manchester
Wander Lairson Costa
Rok Garbas
Mike Hommey
Ted Mielczarek
Cameron McCormack
Jeremy Chen
Sebastian Hengst
Ryan VanderMeulen
Wes Kocher
Manish Goregaokar