When running setup_packages in a docker image that derives from another,
we're currently overwriting the file that contains the apt sources for
the package artifact repositories that were used for the parent docker
image.
This doesn't cause practical problems for the existing docker images,
but in some cases where a user gets a one-click loaner, it might cause
problems when they try to install a package that has a dependency that
can't be fulfilled once those sources are overwritten.
To give a practical example, installing the gdb package from wheezy
requires libpython2.7, but if you try to do that on a derivative of the
debian7-base image, you don't have the deb7-python artifact repository
in your sources.list, and would fail to install gdb because apt can't
install a version of libpython2.7 that can be installed alongside the
python2.7 that is installed.
By putting easy repository in a separate file, named after the task id
of the corresponding package task, we ensure each an every one of them
is uniquely represented in /etc/apt/sources.list.d.
--HG--
extra : rebase_source : efb83b8c292a28c43ede24d5a3879dfbbfe94af7
This will disable libxss-based code in the screenshot tool, like it was
before the switch to Debian-based build images in bug 1399679. There are
other places in the code that use libXss, but they don't require the
development headers, and use dlopen().
--HG--
extra : rebase_source : 4842361b680c70924b9fc0890a7e4f4dcbc7e338
This is basically the same deal as e331a3b9fae2. Caching is hard.
MozReview-Commit-ID: 9uWHHdnHgq1
--HG--
extra : rebase_source : fc7107a018f067d83dbee3796d8861c5187823b2
We start from the image used for Firefox builds, and add the debug
packages for all the system libraries.
--HG--
extra : rebase_source : 2c759975d9837beabdc08a15fd926a99fd1cecf8
The binutils we currently use as part of our GCC toolchain artifact
doesn't understand some relocations in the CRT objects on Debian
stretch, making the embedded CRT objects from bug 1427344, which we want
to remove in bug 1431251, necessary.
OTOH, there is no benefit from using our GCC toolchain artifact for host
compilations on those builds. In fact, Android builds, which are in a
similar position, being built on Debian stretch and being cross-builds,
don't care to use our GCC toolchain artifact.
It's arguably a good thing that those builds are not tied to the version
of GCC we use to build Firefox for linux, so let's remove this
dependency.
--HG--
extra : rebase_source : a80d4e4fb01a4862b844ebde0c521a635f462c0a
I tested this on automation and the build went on, though I couldn't test
the bindgen output because the build right now is busted on one dependent crate
with rust beta, which is the first toolchain that has this package, and will go
to release shortly.
This should work though! If I need more changes I'll adjust them in bug 1432153.
You can test the repackage manually with repack_rust.py --toolchain beta, for
example.
MozReview-Commit-ID: GI2f6vGVqTe
Don't build ucl when building upx, Debian stretch has a recent enough
version. In fact, the last upstream version doesn't build with GCC in
Debian stretch (http://bugs.debian.org/811707)
--HG--
extra : rebase_source : aae67773b9dd3b99f6ddf9ab7f59a628037e6925
This job requires cmake, which should be fixed, but in the meanwhile,
create a separate docker image with it installed, based on the image we
use for other spidermonkey builds.
--HG--
extra : rebase_source : da43a7999b6bd86dbba816358d907c902415bed4
We've observed apt failures multiple times where it apparently fails to
get a file in full from snapshot.debian.org. Making it retry
automatically rather than retriggering tasks seems better.
--HG--
extra : rebase_source : f3ffb415ccc30b7e7c44e6a48b29eb20e69efdd5
The apt in Debian stretch doesn't allow repositories with a Release file
not being GPG signed. Setting up GPG signatures on the
taskcluster-artifact-based repositories is a tricky process, and not
strictly necessary. It turns out not creating a Release file at all
works just as well, and works across all current Debian versions. The
packages priorities remain the same, such that packages from those
repositories are still prefered over the ones from the main Debian
repository (as long as versions are higher).
See comment in cloud-mirror-workaround.sh for that part.
--HG--
extra : rebase_source : df5af330859a314285a6c1922d899489997d2f19
That image is used to derive all the debian7-* images, and its
definition is parametrized, which will allow to create other images
based on other versions of Debian, from the same definition.
XZ_OPT is kept in each of those because we don't want to automatically
set it in all further derived images.
--HG--
extra : rebase_source : 7f4597c1ea4af83627a9373dbdc7945d20b7d996
The base images from docker hub actually contain a
/etc/apt/apt.conf.d/docker-clean that does the equivalent of an apt-get
clean after installing packages.
--HG--
extra : rebase_source : 190de9e3b10a0309cf9cfb3260a91477a5a93ba3
python-dev was required to build mercurial, but the need for that was
removed in bug 1429669.
The others were required for mingw32 toolchains, but they are using a
different docker image and will switch to another different docker image.
--HG--
extra : rebase_source : b65c586a325f220c565e79afb3d3c9acc9f922bc
So far, the best we've been able to do is to upload an image to the
docker hub, and point an image's Dockerfile's FROM to the version
uploaded onto the hub.
That is a cumbersome process, and makes the use of "layered" docker
images painful.
This change allows to declare a parent docker image in the
taskcluster/ci/docker-image/kind.yml definitions, which will be
automatically loaded before building the image. The Dockerfile can then
reference the image, using the DOCKER_IMAGE_PARENT argument, which will
contain the full image name:tag.
Some details are left off, for now, such as VOLUMEs. At this point,
VOLUMEs should all be defined in leaf docker images.
--HG--
extra : rebase_source : 221cff0ca5a91d694ff5c3626fe707c15ba45e23
Now that `mach taskcluster-build-image` can, we can avoid all the manual
handling based on curl and jq in the image builder.
An additional advantage on relying on `mach taskcluster-build-image`
doing more is that less changes to the build-image.sh script will be
necessary, and thus less updates of the image builder docker image.
--HG--
extra : rebase_source : dd174d60675e41e4391894f28235c674c1840829
This allows to avoid writing out a tar file to then extract it to feed
it to `docker build`. This is essentially what the image-builder docker
image does, except it uses a temporary file for the tar.
--HG--
extra : rebase_source : 8275d737e02714fc198d3ba3d3e62e3f18d8e0bf
While spawning `docker load` is likely to work on developer machines,
on automation, it requires a docker client that is the exact same
version as the server running on the taskcluster worker for
docker-in-docker, which is not convenient. The API required for `docker
load` is rather simple, though, and can be mimicked quite easily.
While this change in itself is not necessary for developer machines,
it will allow to re-use the same command for the image-builder to
load a parent docker images when deriving one from another. We could
keep a code branch using `docker load` but it seems wasteful to maintain
two branches when one can work for both use cases.
--HG--
extra : rebase_source : d72956d7dd329b92564cbaa3fbfe0687d4d5d994
The zstd command we spawn, if available at all, might be the wrong
version: zstd changed its stream format in an incompatible way at some
point, and the version shipped in e.g. Ubuntu 16.04 uses the old format,
while the version taskcluster relies on uses the new format.
Relying on gps's zstandard library allows to ensure we use the right
version. Another advantage is that we can trivially pip install it in a
virtualenv if it isn't available on the system running the command.
If we're ridding ourselves of the subprocess spawning for zstd, we might
as well cover curl as well. Especially considering the error handling
when subprocesses are involved is not trivial, such that the current
error handling code is actually broken and leads to dead-lock
conditions, when, for example, curl is still waiting for the python side
to read data, but the python side is not reading data anymore because
an exception was thrown in the tar reading loop.
--HG--
extra : rebase_source : 054c37cfaa68bf475b37545ebaa99144584b93d4
The used pattern:
except Exception:
error = sys.exc_info()[0]
finally:
...
if error:
raise error
actually loses everything that is interesting about the original
exception. Not catching the exception just makes it thrown up the stack,
except when a different exception is thrown from the finally block,
which is what that if error: raise error is attempting to do... except
it doesn't throw the original exception, but its type only.
--HG--
extra : rebase_source : 17601fcc90fcdfefd93c4267f3cd33425d5326fd
Now that we don't need to read the contents of a file to hash the
contents of a docker image context, we can avoid creating a file
in generate_context_hash.
--HG--
extra : rebase_source : 98abe9bfdc48b612a3d251296991d0f769b449fd
This will allow us, down the line, to avoid creating a file at all in
some cases.
--HG--
extra : rebase_source : e4ea40341836cf24aa6d61c905b2efa660ee13f2
It's been more than two weeks since the 1.23 stable release, and
we're making official builds with that toolchain release, so begin
requiring that version so new language features can be used in
development.
MozReview-Commit-ID: E6WuP41ceTn
--HG--
extra : rebase_source : 75850dd9edbf8e3f9beab394e4af7fad76ce3b17
When we branched to beta, the MinGW build started failing with a strange error
like "could not find the robustcheckout Mercurial extension". We disabled it
for a while, but now we're re-enabling it (and fixing the problem.)
The root culprit of this was that we were using the incorrect mozconfig. MinGW
does all sorts of stuff in the mozconfig, but the beta branch overrides the
mozconfig using platform_overrides in
testing/mozharness/configs/builds/branch_specifics.py
We avoid this override by changing the MinGW platform so it doesn't match
and the mozconfig doesn't get overridden.
MozReview-Commit-ID: JkETWCRHacO
--HG--
extra : rebase_source : 5ef7d96804acd3a4eaa6dd6cc8b483f3de438b61
New Android-Gradle plugins pin the build-tools version, and we want to
be consistent between Gradle and moz.build.
MozReview-Commit-ID: ApWS4rHzPuH
--HG--
extra : rebase_source : 22008e9333b15c594ce26c2a52f67396d6e3ab84
extra : source : f918500d9cf5112b70bc8e0a120df435b02252b7
Turns out Google's maven repository doesn't publish checksums. I
can't imagine why not, but there it is. We have to think more about
whether to trust the artifacts downloaded from maven.google.com.
MozReview-Commit-ID: CdWijorq1IV
--HG--
extra : rebase_source : 6c66cf1444876624f10409ea6437863e2c2ea9b0
extra : source : 0850b319efd43ac8f4d61485451722975da55ca1
New upstream release.
- Avoiding argument copies improves memory footprint.
- RwLock<T> no longer requires T to be Send.
- AsciiExt trait methods are now directly available
on str, [u8], u8, and char types without a `use`
statement.
MozReview-Commit-ID: 7Rx8uoNTMqH
--HG--
extra : rebase_source : 39d6297d61d19d710a1376557e4b4d81bdab02c9
This patch disables the stdcxx-compat check for the sm-fuzzing build which
requires patching autospider as well. Furthermore, it switches the build
to linux64-clang-6-pre because the older clang 3.9 does not support trace-pc
instrumentation. Finally, it excludes fuzzing parts from the vanilla allocation
check.
MozReview-Commit-ID: FdhCIFdUore
--HG--
extra : rebase_source : c41bda01cb42f2ef0cd5a1675d88bdb55d9dc8c9
So we can change a string in this file to force the use of new caches,
thus ensuring a clean break from one changeset to the next.
MozReview-Commit-ID: EZsR23a1PZE
--HG--
extra : amend_source : bf6a3c53ab14f2eb249fdf57ff70dce97388f0a0
Switch almost all builds currently using the desktop-build image to use
the right debian7-*-build image instead. The only exception is the rust
bindgen spidermonkey builds, that require cmake being installed, but I
don't want to add it to the base images because that involves risking a
cmake dependency unwantedly slipping in Firefox (rust-bindgen ironically
requires cmake to build a single C++ file...)
Bump mingw version to get the newest commit and do not include the
un-needed dw-extras.h on MinGW (thanks Jacek!)
MozReview-Commit-ID: OjO93XHCxs
--HG--
extra : rebase_source : 933bbb385004988a23d1069c9cd3241b3a3b336e
It is no longer used as part of the build, now that publishing is handled in
separate tasks.
Differential Revision: https://phabricator.services.mozilla.com/D373
--HG--
extra : rebase_source : a6e31fe5d554076d4da584f0993c21b6c1a8ebd0
This makes the XZ_OPT environment variable set for every task that runs
on those docker images, effectively enabling parallel compression for
all of them.
For e.g. the gcc 4.9 toolchain build task, this makes the final
compression step take 23s instead of > 3 minutes.
--HG--
extra : rebase_source : 83809f1f72007cc0829573f1def29a9afdc300c9
This adds an 'override' for the default scheduling component for tests, which
is based on their suite.
MozReview-Commit-ID: 6vd8sb2zeuU
--HG--
extra : rebase_source : 25988c6790287e01fa7751effa72e8b924858948
llvm-symbolizer is necessary to get symbols in llvm-dsymutil crash
dumps. While we could use the one from clang during the build, it's
better if the llvm-dsymutil toolchain is standalone for local testing.
--HG--
extra : rebase_source : 5cd234a3e14ab52a4ce759821e0e756e68167797
When I originally wrote the llvm-dsymutil build script in bug 1430315,
I wasn't setting CMAKE_BUILD_TYPE to Release, and was ending up with
a very large binary (> 300MB), so I stripped it.
When I later set CMAKE_BUILD_TYPE to Release, I left the manual
stripping on, but that removes symbols that are useful for stacktraces
when dsymutil crashes (the Release type still leaves out debug info).
--HG--
extra : rebase_source : 802daadc24c0090574b1a44ea8b4e6c25735f703
By default, wget prints dots every 1k bytes. This can render a
lot of output for large files. We switch to the "mega" style, which
makes each dot represent 64k, thus reducing output by up to 64x.
We also force the use of dot display. By default, it uses "bar"
which attempts to use terminal formatting if possible. Since most
of this code executes in CI and terminal control characters can
interfere with logged output, we force the use of "dot." (Although
wget appears to automatically switch to dot in TC today. But
consistency is good.)
MozReview-Commit-ID: IpTWJdcauTV
--HG--
extra : rebase_source : 5c9aa1bbdcd78eaa0b31347ad026a2c1beaedc03
Now that we have a version of xz-utils that supports parallel compression,
we're not going to use it.
--HG--
extra : rebase_source : 7ac7840e979176b8d2042a4ef35f5868acd6f224
There were a few constraints in the choice of the version of dpkg to
backport:
- 1.17.20 is the first version that supports the debian source format
for that xz-utils package.
- versions >= 1.17.10 and <= 1.17.22 fail to build on wheezy.
- versions >= 1.17.21 depend on a version of patch not available on
wheezy.
All in all, the simpler choice was to go with version 1.17.20 with a
backport of the build failure fix.
That version of dpkg breaks the version of devscripts in wheezy, so the
version from wheezy-backports would be better to use, but we can't
unconditionally use it on all builds, because it happens that
mk-build-deps from that version is broken with the dpkg in wheezy.
In the end, it's simpler to build that backport and rely on package task
dependencies rather than selectively install the package from
wheezy-backports, so we do that. Except we can't use version
2.14.11~bpo70+1 because of bug 1419577.
--HG--
extra : rebase_source : 19ad1a44b770229fbc7e15bbcf01d3cb101315a8
In some cases, apt-get can't figure out how to install build
dependencies, and aptitude, with hints, is necessary instead. But
aptitude also doesn't necessarily work in some cases (e.g. it picks the
wrong automake version to build valgrind, although that's a problem in
valgrind's build dependencies). All in all, it's still better to use
apt-get for all the simple cases.
--HG--
extra : rebase_source : 5dcde3507f2642be8f284b6148d1ccaa4761164a
It can be necessary to use the artifacts from package tasks as build
dependencies for other package tasks.
--HG--
extra : rebase_source : a625670ae7af9a99c848924bdba9368c556e6766
Not all changes to debian_packages.py lead to actual changes to the
corresponding tasks. And since the tasks are pretty much self-contained,
we can consider that variations of them will be entirely represented in
the command used for the task. The only exception is the patch content
when there is an associated patch.
--HG--
extra : rebase_source : e70fa41a824106b5ceca09fd535c0c36bb0739ac
In many cases, building docker images starts on machines that don't have
a cached checkout, and it often takes forever to get a full clone. It
used to be worsened when 3 jobs could run at the same time because the
worker would start up clean, and 3 jobs would be doing a mercurial clone
at the same time, thrashing I/O, but that part is fortunately fixed.
It is still, however, appreciable not to waste time in the mercurial
clone part of image creation.
--HG--
extra : rebase_source : 8c76bc91e1d5102f68c43e1050d61971fef32e9f
The image builder image we use to build docker images is updated
manually, and not necessarily when changes occur in tree that should be
reflected by a new image builder image. For instance, its run-task is
currently outdated. Not enough that it's actually a problem, but it
could rapidly become a problem.
There is also a lot of friction when trying to make changes in how
docker images are built, and while last time I tried, I ended up not
being able to do the changes I wanted to make because the docker version
on the host is too old, but this is already the second time I've been
trying to make things better and hit a wall because the the image
builder is essentially fixed in stone on the docker hub.
So with this change, we make all the docker images use the in-tree image
builder image, except itself, obviously. That one uses the last version
that was uploaded. We may want to update it at some point, but not doing
so will only impact building the image builder image itself, not the
other ones.
--HG--
extra : rebase_source : 978cf033732cbbbb277d206dec69660175b82afa
Back in bug 1360609, we added `run-on-projects` to a list so that the
toolchain tasks wouldn't run on every push on release branches.
Fast forward to now, and they're depended upon by other tasks, meaning
they are triggered when appropriate, without resorting to that trick. In
fact, the commit message for bug 1360609 said we could switch to an
empty list once the jobs have dependencies.
The same is true from package tasks, which, in fact, I suspect would
happen on every push on release branches.
The only exception is for a few toolchains that are depended upon by
nothing, and that are produced for developer consumption with e.g. mach
artifact toolchain.
--HG--
extra : rebase_source : bb8624fed7490b85f4bd72b7ceb2db7a72b4c2ab
Mike Hommey
Narcis Beleuzu
Cosmin Sabou
shindli
Ben Hearsum
Tom Prince
Mark Banner
Joel Maher
Aki Sasaki
Kim Moir
Rob Wood
Simon Fraser
Gurzau Raul
Gregory Szorc
Marco Castelluccio
Nick Thomas
Geoff Brown
Tom Ritter
Tiberius Oros
Andreea Pavel
Brindusan Cristian
Sylvestre Ledru
Emilio Cobos Álvarez
Dorel Luca
Ralph Giles
Nick Alexander
Margareta Eliza Balazs
Andrew Halberstadt
Csoregi Natalia
Johan Lorenzo
Jesse Schwartzentruber
Michael Kaply
Christian Holler
Dustin J. Mitchell
Ciure Andrei
Kartikaya Gupta