There are e.g. some build infrastructure changes that we want to have a
controlled impact on the Firefox builds we produce. We have, in multiple
occasions, gone through manual work to compare Firefox builds, most of
the time using the diffoscope tool (https://diffoscope.org/).
This change introduces a new task kind that takes two Firefox builds as
input, either by name (reference to a build from the current task graph)
or by index (reference to a build from a previous push), and compares
them.
In order to get a Firefox build by index, we rely on dummy tasks with
an optimization we expect to always hit, so we add the necessary bits
to ensure those dummy tasks can go through up to the optimization phase
and be optimized out there.
--HG--
extra : rebase_source : 37482f67652dab2fcef2db4e6b8efe653999bae5
The spidermonkey mozjs and rust-bindings builds run sed on
$topsrcdir/.cargo/config.in to generate the cargo config they use, but
they previously only replaced the @top_srcdir@ substitution. This patch
makes them replace any other substitutions with an empty value to add
a bit of future-proofing.
MozReview-Commit-ID: 1DzP9vXxHMD
--HG--
extra : rebase_source : e8c0268a2a6e91ca2000b340beee2dcff0636591
We currently use a 32-bit Rust toolchain for win32 builds, but this can lead
to OOM situations. This patch makes win32 builds use a 64-bit Rust toolchain,
which requires a little bit of extra configuration because rustc needs to
be able to find a link.exe that produces 64-bit binaries for building
things like build scripts, which are host binaries.
We will now generate a batch file that sets LIB to the paths to 64-bit
libraries and invokes the x64-targeting link.exe, and add a section to the
.cargo/config file to instruct cargo to use that batch file as the linker
when producing 64-bit binaries.
MozReview-Commit-ID: 9vKBbm7Gvra
--HG--
extra : rebase_source : 599b3b661c7a8a5db1f32a2a9732fc202fb55e1e
Add a cross-compilation copy of rust's standard library targeting
i686-pc-windows-msvc to the win64-rust toolchain package so it
can be used to build for win32 as well.
MozReview-Commit-ID: 3598VZrDjIH
--HG--
extra : rebase_source : f1b25a68a67ae7f9c505a42d17f29dbedf59a49d
Instead of duplicating Dockerfiles between taskcluster/docker/*
directories, which can be error prone for very close images, it can be
desirable to use the same file. This change allows to set the
`definition` keyword on a docker image definition in kind.yml that
will make the task use the files from taskcluster/docker/<definition>
instead of taskcluster/docker/<image_name>.
--HG--
extra : rebase_source : 11ae231f66ca6a77896c1cff6c1580d04210f052
Ideally, we'd simply use the --build-arg docker argument along with ARG
in the Dockerfile, but that's only supported from Docker API 1.21, and
we're stuck on 1.18 for the moment.
So we add another hack to how we handle the Dockerfile, by adding a
commented syntax that allows to declare arguments to the Dockerfile.
The arguments can be defined in the docker images kind.yml file through
the `args` keyword. Under the hood, they are passed down to the docker
image task through the environment. The mach taskcluster-build-image
command then uses the corresponding values from the environment to
generate a "preprocessed" Dockerfile for its context.
--HG--
extra : rebase_source : 26a43dd680c1ab97b1a4689a23c55594a3b21b67
Landings of e.g. bug 1427336 triggered new toolchain jobs. One of those
jobs, because of wrong changes in bug 1421100, downloaded a new rust
compiler beta instead of the intended fixed beta version. In turn, that
new rust compiler beta fails to compile the slog crate.
Now, because of how toolchain cache indexes work, every new win32 job
picks that new unintended rust compiler beta version, even on branches
where 1427336 hasn't landed.
I couldn't find a way to force the right beta version, so we're pretty
much stuck with that toolchain index pointing to the wrong version of
rust beta.
By backing out bug 1426324, we return to a toolchain index that is known
to have an artifact for the right rust compiler beta.
Unfortunately, if something triggers a new TW32(rust) job after this,
that toolchain index will be busted as well.
Giving a directory to %include would copy all leaf files under one
single directory in the context image. The only image affected is
valgrind-build, which ended up having a dot-config/pip.conf file instead
of dot-config/pip/pip.conf, meaning valgrind jobs weren't using the
pip config.
--HG--
extra : rebase_source : 518c8ca1617b57ae4b4bb83a85340de5515c26c5
Android cppunit, test-verify, and mochitest-gpu have been running on lower end
aws instances. It is probably better to run all emulator tasks on xlarge.
libcrypto, part of OpenSSL, and that dmg links against, has a varying
ABI, and something built against libcrypto on Centos won't run on Debian
and vice versa. It might not even work between versions of the same OS
(e.g. Debian 7 vs. Debian 9).
Because of that, it is desirable to statically link it.
This incorporates https://github.com/mozilla/libdmg-hfsplus/pull/1
and sets OPENSSL_USE_STATIC_LIBS when building libdmg-hfsplus.
--HG--
extra : rebase_source : 21a46f707494f388a899c08d0923f8b393d12cd1
fontconfig uses expat by default to read its xml configuration, but when
expat is not there at build time, it falls back to libxml2. We ended up
in this situation, while on Debian, fontconfig is built against expat.
This makes no practical difference, since we're not actually using
fontconfig, but for some reason, the difference in dependencies has an
impact on how ld chooses to arrange the .plt and .got.plt sections,
meaning that even though the code and data is originally identical, in
the .o files, the resulting linked machine code is largely different
because of all the applied relocations changing the offsets of e.g. call
instructions for function calls through the .plt. This results in large
differences in .plt, .init, .text, .data.rel.ro, as well as symbols list
when building on Debian.
This thus is meant to help make the differences more tractable.
--HG--
extra : rebase_source : 7a731c34074a50e84412f73ab9499248345fb14a
OSX (cross) repackages are currently using a tooltool manifest to get
libdmg and hfsplus. Change those jobs to use the toolchain artifacts
instead.
At the same time, modify the repackage mozharness script's _run_tooltool
so that it doesn't fail with MOZ_TOOLCHAINS being set but without a
tooltool_manifest_src, matching the similar function in buildbase.py.
--HG--
extra : rebase_source : d128d4709c5d1d28d1a6b9c585fde82e99f725c7
I suppose it was setup through ~worker/.hgrc before we started
installing a /etc/mercurial/hgrc that enables a few other extensions
and sets some preferences.
There is no reason to now have two places where mercurial is being set
up, and it feels natural that we set it up at the system level.
Ideally, we'd also clean up the centos6-based images, but they require
an update of the centos6-build and centos6-build-upd images on the
docker hub, which is not really convenient, and those images are going
to be obsoleted soon anyways (bug 1399679).
--HG--
extra : rebase_source : 32c9cdf5d0fe8ac2c60a1c5a38e572c83a4783b2
Bug 1389715 removed the image definition in taskcluster/ci/docker-image
as well as the files associated with it in
taskcluster/docker/desktop-test, but the Dockerfile in there was the
only use of the Ubuntu 12.04 setup script, so it is currently unused.
--HG--
extra : rebase_source : 7d8018e7c94e2625ff9822a2d66231722a030394
The debian-build-system-setup.sh script doesn't use the install-make and
install-cmake scripts, so it's unnecessary to install them in /setup.
--HG--
extra : rebase_source : 4ba24b9827e67b9c7ad203e789e00e19d37786da
The script was added in bug 1179893 but looks like it has actually never
been used. A duplicate of the file was used for the upload-symbols
image, but that was removed in bug 1422740.
Since it was the only file in desktop-build/bin, we stop copying the
directory in docker images.
--HG--
extra : rebase_source : 4bcdb5ba0118e87455c6f596bf54e4528fe1b1ef
While we're here, add a missing prepare_vcs_checkout for the
comm-central checkout.
--HG--
extra : rebase_source : 788a288330e34b5551ec2b12726a755e268566c2
It turns out that in all cases it was the last tooltool manifest entry,
so we can remove the tooltool manifests entirely, and remove all
references to them.
--HG--
extra : rebase_source : d8447b5422e63e88444008fddb76d658829694de
We're about to remove some tooltool manifests, so we need those calls to
work properly when TOOLTOOL_MANIFEST is not set.
--HG--
extra : rebase_source : 89d41021a87915dc9133e61543352e3bda1dace4
Back when we started needing gtk+3 to build Firefox, we were using mock
to setup the build environment, and a tooltool package was the most
sensible way to handle this.
Fast forward to today, and we're close to moving the build environment
to Debian, which comes with gtk+3 packages. But in order to simplify
the various checks for the transition, it is desirable to stop using the
tooltool package. Which we can actually do in a reasonable way now that
we use docker images instead of mock, by building and installing gtk+3
in the build environment images.
So we modify the script that was producing the gtk+3 tooltool packages
such that it installs gtk+3 in the docker images, both 32 and 64 bits.
And invoke it when creating the desktop build environment docker images.
--HG--
extra : rebase_source : 75e987d6de7f3ae8a3d9b478fc173e191d28aace
It turns out that in all cases it was the last tooltool manifest entry,
so we can remove the tooltool manifests entirely, and remove all
references to them.
--HG--
extra : rebase_source : 0aa9ef8151c2fccf62507dfecc0bc57b157772e1
We're about to remove some tooltool manifests, so we need those calls to
work properly when TOOLTOOL_MANIFEST is not set.
--HG--
extra : rebase_source : 38ca0e3b894097ed3667901b05af79062a6c82c2
Back when we started needing gtk+3 to build Firefox, we were using mock
to setup the build environment, and a tooltool package was the most
sensible way to handle this.
Fast forward to today, and we're close to moving the build environment
to Debian, which comes with gtk+3 packages. But in order to simplify
the various checks for the transition, it is desirable to stop using the
tooltool package. Which we can actually do in a reasonable way now that
we use docker images instead of mock, by building and installing gtk+3
in the build environment images.
So we modify the script that was producing the gtk+3 tooltool packages
such that it installs gtk+3 in the docker images, both 32 and 64 bits.
And invoke it when creating the desktop build environment docker images.
--HG--
extra : rebase_source : fe18bfb2ec8db183c44838d5a7a0051322b2a9c0
On Debian, the autoconf binary is autoconf2.13 while it is autoconf-2.13
on Centos.
In make-source-package.sh, we need to run autoconf to generate the
old-configure to include in the package, so try both.
In hazard-analysis.sh, we actually don't need autoconf itself, so just
copy configure.in to configure.
--HG--
extra : rebase_source : d21075394c69cd7cd6738da645173eb29f4a1259
Note that we need to use the virtual-with-gpu instances on windows for
WebRender to even start up.
MozReview-Commit-ID: 6fMDun7casP
--HG--
extra : rebase_source : 5068bd17d11725c2c0f5bd0b387a54047475f0c6
For example, jittests will be an inclusive test suite; all files which might
affect the suite are tagged with
SCHEDULES.inclusive += ['jittest']
but those files usually also schedule all of the exclusive components (including
the platform families android, linux, macosx, and windows). This makes sense:
those files could potentially affect any other test suite on any platform too.
But the jittest job on Android, for example, needs to run only if the jittest
component is scheduled -- it does not need to just because something
Android-related changed. So its optimization should be {skip-unless-schedules:
['jittest']}, not {skip-unless-schedules: ['jittest', 'android']}.
This fix "figures out" the distinction by looking at what kind of component the
test suite is. Maybe that is too magic, and we should also have to write
"component: implicit" in the tests/*.yml file.
MozReview-Commit-ID: EIsVvi1vziE
--HG--
extra : rebase_source : eb7ad26db801028dc514af6c2eaaadb649445db0
We build packages of the same versions that were installed by
taskcluster/docker/recipes/install-cmake.sh and
taskcluster/docker/centos6-build/system-setup.sh in the desktop-build
image.
--HG--
extra : rebase_source : 843b89065daabd450f54ebf7a2cf55d00977e23a
This also renames the existing test sets for qr to be linux-specific, so
we can have a different test set for windows QR builds. The windows10-64-qr
gpu mochitests will run on all nightly branches (so inbound, autoland, m-c,
try) by default.
MozReview-Commit-ID: F2NjCTHYg13
--HG--
extra : rebase_source : eb107b11d995a84bd76885e1af241ca05f634684
Previously we had linux64-qr as the only QuantumRender test platform.
Soon we will have windows10-64-qr as well and (eventually) we will have
some macOS -qr tests as well. So this patch generifies the existing
regexes to match these platforms.
In a couple of places redundant platform matching lines were removed, to
avoid the case where a given platform (e.g. windows10-64-qr) matches
multiple regexes (e.g. .*-qr/.* and windows.*) which produces an error.
MozReview-Commit-ID: 8YO9lQETVYM
--HG--
extra : rebase_source : 60b59fedd7cab71f7cf2118feea16b058bd4654c
We currently use a 32-bit Rust toolchain for win32 builds, but this can lead
to OOM situations. This patch makes win32 builds use a 64-bit Rust toolchain,
which requires a little bit of extra configuration because rustc needs to
be able to find a link.exe that produces 64-bit binaries for building
things like build scripts, which are host binaries.
We will now generate a batch file that sets LIB to the paths to 64-bit
libraries and invokes the x64-targeting link.exe, and add a section to the
.cargo/config file to instruct cargo to use that batch file as the linker
when producing 64-bit binaries.
MozReview-Commit-ID: 9vKBbm7Gvra
--HG--
extra : rebase_source : 366dd966cafe4f07b8e59fc170d2db2dada32627
Add a cross-compilation copy of rust's standard library targeting
i686-pc-windows-msvc to the win64-rust toolchain package so it
can be used to build for win32 as well.
MozReview-Commit-ID: 3598VZrDjIH
--HG--
extra : rebase_source : baab6d8718d7a8d38a353a2bffcea14dcee45c8f
The "contract" for toolchains is that extracting foo.tar.xz creates a
directory named foo/. That is however not true for mingw32.tar.xz, which
extracts into gcc/, possibly overwriting files from the gcc.tar.xz
archive (which is also used for mingw builds, for the host part).
This is also not true for nsis.tar.xz, but it reportedly has problems
when it's not in the same directory as mingw32.
But mingw32 doesn't actually need to be mixed with gcc, so it's better
to separate them as they are supposed to be.
--HG--
extra : rebase_source : 30d90af64459bbb31bc076e48f3c661fa9cd4a79
f179a112278d (bug 1373878) established tasks for Rust tests. It
created new Treeherder "platforms" for each task.
These platforms (which still only have a single task) seem wasteful.
Let's remove the one-off Treeherder platform and move the "rusttests"
tasks into an existing platform so Treeherder's output is more
concise.
MozReview-Commit-ID: 8Fcph0r5wad
--HG--
extra : rebase_source : 3035d0ea50208911440498a108f653c298903352
The symbol-upload task currently downloads the symbols-full.zip artifact
from the build task and then uploads it to the symbol server. These zip
files can be very large (>1GB) so we spend a lot of time doing that.
Now that we're uploading to Tecken instead of Socorro, we can instead
just send the URL of the artifact to Tecken's upload API and ask it to
fetch that directly:
https://tecken.readthedocs.io/en/latest/upload.html#upload-by-download-url
This should make the symbol upload task a fair bit faster.
MozReview-Commit-ID: 8HcbgrWYT1O
--HG--
extra : rebase_source : 4e8f7a28c956befb3e291e8be4d41a2b6728e5cd
Building for some tier-3 platforms imply building without a JIT, and it
happens quite regularly that this setup is broken by API changes in
Spidermonkey.
This adds a new job with JIT disabled, but skip tests for now because
some fail or crash.
--HG--
extra : rebase_source : 3c6e1dfb3cd7d0bff59c494f6230c9f1b55479ed
Adjust post-beetmover-dummy's tasks and deps.
- Fennec doesn't have beetmover-repackage or beetmover-checksums,
so add `beetmover` to the post-beetmover-dummy kind-dependencies.
- Add a fennec-promote post-beetmover-dummy job.
- Remove the extraneous -ship post-beetmover-dummy jobs. Once we
removed the assumption that dummy jobs had to be in the same phase,
these became redundant.
In testing, this looks good. For the next step, we may want to split
these dummy tasks up by `build_platform`. Then downstream tasks could
then optionally filter their dummy deps by `build_platform`; this
would allow for certain platforms to proceed on to the next steps
sooner, rather than wait for the slowest platform to finish.
I also suspect we don't need post-beetmover-checksums-dummy at all;
it's redundant.
MozReview-Commit-ID: EeHjwTQnVB1
--HG--
extra : rebase_source : 812288cf083499d38e3e47a203c43163afd8e2a5
extra : source : e78626133e88e124922a43b5af7ebfd5e5325360
`taskgraph.create` can add an additional dependency to tasks prior to task submission. The queue will error out if we submit a task with over 100 dependencies, so we should limit ourselves to 99 dependencies here.
MozReview-Commit-ID: ClT0vjYBPp4
--HG--
extra : rebase_source : e1f168a5c472be8d45e689517fff0a47ba1bbe7c
extra : source : 351a176ad1181d9a43056098a66b80f1fa56e401
..and remove support for when.files-changed in the test kind. It is still used
for other kinds, and that will be addressed in other bugs.
This is re-landing of this bug, now without running test-verify excessively.
MozReview-Commit-ID: GBilXAktICZ
--HG--
extra : rebase_source : 6cc9a3b5a365d74689946bfa0296f51bc08c2113
libcrypto, part of OpenSSL, and that dmg links against, has a varying
ABI, and something built against libcrypto on Centos won't run on Debian
and vice versa. It might not even work between versions of the same OS
(e.g. Debian 7 vs. Debian 9).
Because of that, it is desirable to statically link it.
This incorporates https://github.com/mozilla/libdmg-hfsplus/pull/1
and sets OPENSSL_USE_STATIC_LIBS when building libdmg-hfsplus.
--HG--
extra : source : 78f2064b3811db58b364c32ce9b58a3f2dcaf8f8
There is no /lib64 on Debian. OTOH, one doesn't need to give the full
path to a system library in LDFLAGS, so just use -l syntax instead.
--HG--
extra : rebase_source : b795f97ab209499824afa5ef1aee9da52657ceb9
Johan Lorenzo
Mike Hommey
Gregory Szorc
Tom Prince
Chris AtLee
Ted Mielczarek
Ralph Giles
Margareta Eliza Balazs
Coroiu Cristina
Geoff Brown
Andreea Pavel
Ben Hearsum
Kartikaya Gupta
Dustin J. Mitchell
Csoregi Natalia
Connor Sheehan
Rob Wood
Aki Sasaki
Dorel Luca
Simon Fraser
Steve Fink
Gurzau Raul
Rail Aliiev
Kim Moir
Ting-Yu Chou
Narcis Beleuzu
Rok Garbas
Matthew Noorenberghe