In many cases, building docker images starts on machines that don't have
a cached checkout, and it often takes forever to get a full clone. It
used to be worsened when 3 jobs could run at the same time because the
worker would start up clean, and 3 jobs would be doing a mercurial clone
at the same time, thrashing I/O, but that part is fortunately fixed.
It is still, however, appreciable not to waste time in the mercurial
clone part of image creation.
--HG--
extra : rebase_source : 8c76bc91e1d5102f68c43e1050d61971fef32e9f
The image builder image we use to build docker images is updated
manually, and not necessarily when changes occur in tree that should be
reflected by a new image builder image. For instance, its run-task is
currently outdated. Not enough that it's actually a problem, but it
could rapidly become a problem.
There is also a lot of friction when trying to make changes in how
docker images are built, and while last time I tried, I ended up not
being able to do the changes I wanted to make because the docker version
on the host is too old, but this is already the second time I've been
trying to make things better and hit a wall because the the image
builder is essentially fixed in stone on the docker hub.
So with this change, we make all the docker images use the in-tree image
builder image, except itself, obviously. That one uses the last version
that was uploaded. We may want to update it at some point, but not doing
so will only impact building the image builder image itself, not the
other ones.
--HG--
extra : rebase_source : 978cf033732cbbbb277d206dec69660175b82afa
Back in bug 1360609, we added `run-on-projects` to a list so that the
toolchain tasks wouldn't run on every push on release branches.
Fast forward to now, and they're depended upon by other tasks, meaning
they are triggered when appropriate, without resorting to that trick. In
fact, the commit message for bug 1360609 said we could switch to an
empty list once the jobs have dependencies.
The same is true from package tasks, which, in fact, I suspect would
happen on every push on release branches.
The only exception is for a few toolchains that are depended upon by
nothing, and that are produced for developer consumption with e.g. mach
artifact toolchain.
--HG--
extra : rebase_source : bb8624fed7490b85f4bd72b7ceb2db7a72b4c2ab
There are now only a handful of buildbot jobs remaining and the concern over
outdated treeherder exclusion profiles has largely been resolved.
This does remove the tc() group from a substantial number of tasks which will
now show up as top level tasks, potentially adding clutter. In some cases, we
might want to re-add a new group (e.g group builds or compiled tests together).
However rather than try to predict the best group names for tasks I'm unfamiliar
with, I think it's best to land this as is. Then if things are looking too
cluttered at the root namespace, file follow-up bugs as needed.
MozReview-Commit-ID: 8SMwjDwAOzV
--HG--
extra : rebase_source : 2f6d89d11c139bdcd404e7537db799d0e36ee4c3
When trying to remove an ubiquitous group like tc(), it's hard to tell where the
error was located without grepping my filesystem. This makes it a bit easier to
find and fix these errors.
MozReview-Commit-ID: 8NjvB5zOoqb
--HG--
extra : rebase_source : 167d3097f96548cf9c13b602d7d485cb69d48c2d
Relying on the various transforms setting it manually is error prone,
and, in fact, is why bug 1430037 busted beta. This change makes this
setting happen at a single place. This yields the same full task graph
as before, except for *more* chain-of-trust inputs being set now: they
were missing for toolchain tasks (which makes us closer to bug 1384430).
--HG--
extra : rebase_source : b6bf3a3b6da7174957c4c6814b853a51ee8a1e27
The mozharness transform is supposed to set the docker image to
desktop-build when not already set, but was not doing it properly.
I guess this is why some jobs were setting the image themselves, despite
using the mozharness transform.
Consequently, don't manually set the image to desktop-build when it's
the default.
--HG--
extra : rebase_source : 024bd10960bedaee3416785348a5c12498c5286f
taskcluster/taskgraph/transforms/task.py sets an expiry to 28 days for
tasks on try, vs. 1 year on other projects, but only do that when an
expiry is not already set, which docker images do. And they do always
set to 1 year.
But it doesn't make sense to keep large docker images from try for a
year. So use the same retention policy as the default one. We /could/
just remove the expiry from docker images and get the task.py default,
but it seems like whatever future change might happen to that default
shouldn't affect docker images, so it's better to duplicate the setting.
--HG--
extra : rebase_source : b5b46ca34a40ac82c5403b67d5b1aacf8cf8cceb
Build dependencies won't be installed from backports unless they are not
satisfiable in the given Debian release. This is useful to get dh-python
on Wheezy.
--HG--
extra : rebase_source : 1f249b4ceae4fdd9ea37e9a9b9e9b62b48a1c9ed
In the case of mercurial, we don't want to use a .dsc as the original
source, but rather use the debian packaging scripts available in the
upstream mercurial tarball.
--HG--
extra : rebase_source : ec5b288f3994bc0bc1ec9ebce40def807bb7681f
The taskgraph.util.schema.check_schema function validates key names used
in schemas, ensuring they are dashed lower-case. However, it currently
assumes keys are either direct strings, Required or Optional entries,
and either ignores or fails to recognize other patterns.
For example, it ignores Any, and fails to recognize combinations like
Required(Any(...)), which we're going to use in next patch.
--HG--
extra : rebase_source : 4f6ff51a4a9dc9c7d9b6d070e03c6cc6e1befe80
In many cases, building docker images starts on machines that don't have
a cached checkout, and it often takes forever to get a full clone. It
used to be worsened when 3 jobs could run at the same time because the
worker would start up clean, and 3 jobs would be doing a mercurial clone
at the same time, thrashing I/O, but that part is fortunately fixed.
It is still, however, appreciable not to waste time in the mercurial
clone part of image creation.
--HG--
extra : rebase_source : bbe8b001849e59bb655bb0e9766a6071ad38a52c
The image builder image we use to build docker images is updated
manually, and not necessarily when changes occur in tree that should be
reflected by a new image builder image. For instance, its run-task is
currently outdated. Not enough that it's actually a problem, but it
could rapidly become a problem.
There is also a lot of friction when trying to make changes in how
docker images are built, and while last time I tried, I ended up not
being able to do the changes I wanted to make because the docker version
on the host is too old, but this is already the second time I've been
trying to make things better and hit a wall because the the image
builder is essentially fixed in stone on the docker hub.
So with this change, we make all the docker images use the in-tree image
builder image, except itself, obviously. That one uses the last version
that was uploaded. We may want to update it at some point, but not doing
so will only impact building the image builder image itself, not the
other ones.
--HG--
extra : rebase_source : 73e8fc51ea53af1e647fc1d5093c67d614dd009e
This boils down to always setting worker.env to avoid a KeyError.
MozReview-Commit-ID: 1s4az9BFcc2
--HG--
extra : rebase_source : dabed2dedb00d176b829c6c0ff911e0236c5dec4
When looking for perfherder data collection duplicates, we currently
keep full job objects references, which are then used in case an error
occurs, to display the job names of the duplicates.
But those job objects are yielded and may be modified by other
transforms, and presently, by the time a duplicate is found, the
corresponding job object has been modified such that it has no 'name'
key anymore, leading to a KeyError exception when trying to display
the duplicate error message.
So instead of keeping the job objects, which can change, and which we
don't have a real use for, just keep the job name.
--HG--
extra : rebase_source : 204e90a6fe1e4ce62f361451e1176d3195a3383b
In bug 1427326, we added package tasks that can be depended upon by
docker image tasks. In that case, we add the routes containing a digest
for those package tasks when computing the docker image digests.
The problem is that those routes start with 'index.gecko.cache.level-n'
where n varies between try and e.g. mozilla-central. That means the
digest for those docker images varies between try and e.g.
mozilla-central, which then prevents try from using the cached versions
for mozilla-central when there is one, like for other docker images
without package dependencies.
What we really need from those routes is the digest part, which is
independent of the level, and we don't actually care about anything else
in the route string, so just use the digest.
--HG--
extra : rebase_source : 4aecf8472306963da34f2bd4d92675962c0432bc
This was useful when we still had buildbot-based build jobs, but all
it achieves nowadays is add friction when adding new build jobs on
taskcluster.
--HG--
extra : rebase_source : aa6a21a875eff1888c16900acf6d01ff37ab832b
There are e.g. some build infrastructure changes that we want to have a
controlled impact on the Firefox builds we produce. We have, in multiple
occasions, gone through manual work to compare Firefox builds, most of
the time using the diffoscope tool (https://diffoscope.org/).
This change introduces a new task kind that takes two Firefox builds as
input, either by name (reference to a build from the current task graph)
or by index (reference to a build from a previous push), and compares
them.
In order to get a Firefox build by index, we rely on dummy tasks with
an optimization we expect to always hit, so we add the necessary bits
to ensure those dummy tasks can go through up to the optimization phase
and be optimized out there.
--HG--
extra : rebase_source : 37482f67652dab2fcef2db4e6b8efe653999bae5
Instead of duplicating Dockerfiles between taskcluster/docker/*
directories, which can be error prone for very close images, it can be
desirable to use the same file. This change allows to set the
`definition` keyword on a docker image definition in kind.yml that
will make the task use the files from taskcluster/docker/<definition>
instead of taskcluster/docker/<image_name>.
--HG--
extra : rebase_source : 11ae231f66ca6a77896c1cff6c1580d04210f052
Ideally, we'd simply use the --build-arg docker argument along with ARG
in the Dockerfile, but that's only supported from Docker API 1.21, and
we're stuck on 1.18 for the moment.
So we add another hack to how we handle the Dockerfile, by adding a
commented syntax that allows to declare arguments to the Dockerfile.
The arguments can be defined in the docker images kind.yml file through
the `args` keyword. Under the hood, they are passed down to the docker
image task through the environment. The mach taskcluster-build-image
command then uses the corresponding values from the environment to
generate a "preprocessed" Dockerfile for its context.
--HG--
extra : rebase_source : 26a43dd680c1ab97b1a4689a23c55594a3b21b67
Mike Hommey
Andrew Halberstadt
Cosmin Sabou
Tom Prince
Aki Sasaki
Dorel Luca
Dustin J. Mitchell
shindli
Mihai Tabara
Sebastian Hengst
Ben Hearsum
Margareta Eliza Balazs
Johan Lorenzo
Andreea Pavel
Chris AtLee