We have multiple source checkouts. --sparse-profile is ambiguous
as to which one it could refer to. Let's rename the argument so it
is prefixed with the repo/project we are checking out.
Differential Revision: https://phabricator.services.mozilla.com/D13814
--HG--
extra : moz-landing-system : lando
We now have multiple things we may check out. "vcs" meaning "gecko"
is not obvious. Let's change the terminology to be more specific.
Differential Revision: https://phabricator.services.mozilla.com/D13813
--HG--
extra : moz-landing-system : lando
We have multiple source checkouts. --sparse-profile is ambiguous
as to which one it could refer to. Let's rename the argument so it
is prefixed with the repo/project we are checking out.
Differential Revision: https://phabricator.services.mozilla.com/D13814
--HG--
extra : moz-landing-system : lando
We now have multiple things we may check out. "vcs" meaning "gecko"
is not obvious. Let's change the terminology to be more specific.
Differential Revision: https://phabricator.services.mozilla.com/D13813
--HG--
extra : moz-landing-system : lando
This appears to "just work."
While I would like to convert this image to Debian and make it
deterministic, that is more effect than I'm willing to invest at the
moment.
The impetus for this change is unblocking partial clones. Mercurial's
SQLite storage backend apparently hits a SQLite bug in version 3.11
of SQLite (what Ubuntu 16.04 runs) where SQLite complains about
database corruption when there are readers from multiple processes.
Ubuntu 18.04 is running SQLite 3.22 and doesn't exhibit the buggy
behavior.
Differential Revision: https://phabricator.services.mozilla.com/D14228
--HG--
extra : moz-landing-system : lando
This appears to "just work."
While I would like to convert this image to Debian and make it
deterministic, that is more effect than I'm willing to invest at the
moment.
The impetus for this change is unblocking partial clones. Mercurial's
SQLite storage backend apparently hits a SQLite bug in version 3.11
of SQLite (what Ubuntu 16.04 runs) where SQLite complains about
database corruption when there are readers from multiple processes.
Ubuntu 18.04 is running SQLite 3.22 and doesn't exhibit the buggy
behavior.
Differential Revision: https://phabricator.services.mozilla.com/D14228
--HG--
extra : moz-landing-system : lando
For historical consistency and consistency with index paths.
"env_prefix" is no longer used after this change, so it has been
removed.
Differential Revision: https://phabricator.services.mozilla.com/D13876
--HG--
extra : moz-landing-system : lando
We have multiple source checkouts. --sparse-profile is ambiguous
as to which one it could refer to. Let's rename the argument so it
is prefixed with the repo/project we are checking out.
Differential Revision: https://phabricator.services.mozilla.com/D13814
--HG--
extra : moz-landing-system : lando
We now have multiple things we may check out. "vcs" meaning "firefox"
is not obvious. Let's change the terminology to be more specific.
Differential Revision: https://phabricator.services.mozilla.com/D13813
--HG--
extra : moz-landing-system : lando
This uses the latest image_builder image (on docker hub) to build even the
image_builder image.
The change to `docker.py` handles a new API response (`aux`) from the Docker
daemon. It's unclear what this key means, but displaying it is simple.
Differential Revision: https://phabricator.services.mozilla.com/D8441
--HG--
extra : rebase_source : b6a2c2de231bd623521a0a7a0dc595fed059b758
extra : intermediate-source : aa32ba9e6df3e20aab896a63538c8e8a432b4988
extra : source : f33b9e6aca06b7883f048c6c068704680cbfbbb2
This uses the latest image_builder image (on docker hub) to build even the
image_builder image.
The change to `docker.py` handles a new API response (`aux`) from the Docker
daemon. It's unclear what this key means, but displaying it is simple.
Differential Revision: https://phabricator.services.mozilla.com/D8441
--HG--
extra : rebase_source : 2c069da57e416d5e1821e55653d37b23d633ae78
extra : source : f33b9e6aca06b7883f048c6c068704680cbfbbb2
This uses the latest image_builder image (on docker hub) to build even the
image_builder image.
The change to `docker.py` handles a new API response (`aux`) from the Docker
daemon. It's unclear what this key means, but displaying it is simple.
Differential Revision: https://phabricator.services.mozilla.com/D8441
--HG--
extra : moz-landing-system : lando
download-and-compress isn't very complicated and should work on Python 3
with minimal effort. So let's switch it to use Python 3.
MozReview-Commit-ID: 9G1WfcbbKEY
--HG--
extra : rebase_source : 3a6bab06c8500a90413e8b7642a7bf7bdff04a46
Version 0.9.0 bundles a newer version of the zstandard library, which
is a little faster and has a few minor bug fixes (none that we were
likely hitting, however).
MozReview-Commit-ID: 9YgSZ0G41eg
--HG--
extra : rebase_source : 8f5a68323b1e1fe7e9f1dd1a92e132434972d21d
We want Python 3 available everywhere because it is 2018.
MozReview-Commit-ID: L3wufNXKdnp
--HG--
extra : rebase_source : c260923e3c13f8b28e30eaaf6e1bd38f79500052
In preparation for making it usable on Windows, after which point
having it in a directory with "docker" in it doesn't make much sense.
MozReview-Commit-ID: Hgu0buFyJwF
--HG--
rename : taskcluster/docker/recipes/run-task => taskcluster/scripts/run-task
extra : rebase_source : 3c0b502d28b5aad54bd04069efbfda88e25bbb20
So far, the best we've been able to do is to upload an image to the
docker hub, and point an image's Dockerfile's FROM to the version
uploaded onto the hub.
That is a cumbersome process, and makes the use of "layered" docker
images painful.
This change allows to declare a parent docker image in the
taskcluster/ci/docker-image/kind.yml definitions, which will be
automatically loaded before building the image. The Dockerfile can then
reference the image, using the DOCKER_IMAGE_PARENT argument, which will
contain the full image name:tag.
Some details are left off, for now, such as VOLUMEs. At this point,
VOLUMEs should all be defined in leaf docker images.
--HG--
extra : rebase_source : 221cff0ca5a91d694ff5c3626fe707c15ba45e23
Now that `mach taskcluster-build-image` can, we can avoid all the manual
handling based on curl and jq in the image builder.
An additional advantage on relying on `mach taskcluster-build-image`
doing more is that less changes to the build-image.sh script will be
necessary, and thus less updates of the image builder docker image.
--HG--
extra : rebase_source : dd174d60675e41e4391894f28235c674c1840829
In many cases, building docker images starts on machines that don't have
a cached checkout, and it often takes forever to get a full clone. It
used to be worsened when 3 jobs could run at the same time because the
worker would start up clean, and 3 jobs would be doing a mercurial clone
at the same time, thrashing I/O, but that part is fortunately fixed.
It is still, however, appreciable not to waste time in the mercurial
clone part of image creation.
--HG--
extra : rebase_source : 8c76bc91e1d5102f68c43e1050d61971fef32e9f
The image builder image we use to build docker images is updated
manually, and not necessarily when changes occur in tree that should be
reflected by a new image builder image. For instance, its run-task is
currently outdated. Not enough that it's actually a problem, but it
could rapidly become a problem.
There is also a lot of friction when trying to make changes in how
docker images are built, and while last time I tried, I ended up not
being able to do the changes I wanted to make because the docker version
on the host is too old, but this is already the second time I've been
trying to make things better and hit a wall because the the image
builder is essentially fixed in stone on the docker hub.
So with this change, we make all the docker images use the in-tree image
builder image, except itself, obviously. That one uses the last version
that was uploaded. We may want to update it at some point, but not doing
so will only impact building the image builder image itself, not the
other ones.
--HG--
extra : rebase_source : 978cf033732cbbbb277d206dec69660175b82afa
In many cases, building docker images starts on machines that don't have
a cached checkout, and it often takes forever to get a full clone. It
used to be worsened when 3 jobs could run at the same time because the
worker would start up clean, and 3 jobs would be doing a mercurial clone
at the same time, thrashing I/O, but that part is fortunately fixed.
It is still, however, appreciable not to waste time in the mercurial
clone part of image creation.
--HG--
extra : rebase_source : bbe8b001849e59bb655bb0e9766a6071ad38a52c
The image builder image we use to build docker images is updated
manually, and not necessarily when changes occur in tree that should be
reflected by a new image builder image. For instance, its run-task is
currently outdated. Not enough that it's actually a problem, but it
could rapidly become a problem.
There is also a lot of friction when trying to make changes in how
docker images are built, and while last time I tried, I ended up not
being able to do the changes I wanted to make because the docker version
on the host is too old, but this is already the second time I've been
trying to make things better and hit a wall because the the image
builder is essentially fixed in stone on the docker hub.
So with this change, we make all the docker images use the in-tree image
builder image, except itself, obviously. That one uses the last version
that was uploaded. We may want to update it at some point, but not doing
so will only impact building the image builder image itself, not the
other ones.
--HG--
extra : rebase_source : 73e8fc51ea53af1e647fc1d5093c67d614dd009e
Because Ubuntu 16.04 changed in a way that busts gl3 tests and we can't
update the desktop1604-test image anymore.
--HG--
extra : amend_source : bfa07f9f77990dd6915b8c92d218227436bc6fc4
The install-mercurial.sh script currently installs a global mercurial
configuration after installing mercurial manually. In order to share
that configuration with docker images installing a mercurial package
through packages tasks, we move it to a separate file.
We however keep the part setting web.cacerts in install-mercurial.sh,
since it uses a path depending on what kind of environment the script is
run. Moreover, the instructions that come with mercurial to build
Debian packages come with web.cacerts set to the right path already, so
it's not needed in that case anyways.
At the same time, use multiple files in /etc/mercurial/hgrc.d/ instead
of a single /etc/mercurial/hgrc file.
--HG--
extra : rebase_source : 8140d8243cf012489025afe058f467c72224c891
This includes adding TASKCLUSTER_VOLUMES to docker image builds directly. The
env variable is not added as part of the task transform because `run-task` is
not in payload.command. In fact, build-image.sh calls run-task after doing
some other housekeeping.
Ideally image builds would be turned into jobs and all of this would occur
automatically, but that turns out to be quite a bit too complex for this
incidental fix -- perhaps best solved in another bug.
MozReview-Commit-ID: FYHvafJras7
--HG--
extra : rebase_source : 4e3b9ae9900727e7932c13ced34b3f8596d755d9
This includes adding TASKCLUSTER_VOLUMES to docker image builds directly. The
env variable is not added as part of the task transform because `run-task` is
not in payload.command. In fact, build-image.sh calls run-task after doing
some other housekeeping.
Ideally image builds would be turned into jobs and all of this would occur
automatically, but that turns out to be quite a bit too complex for this
incidental fix -- perhaps best solved in another bug.
MozReview-Commit-ID: FYHvafJras7
--HG--
extra : rebase_source : 4e3b9ae9900727e7932c13ced34b3f8596d755d9
run-task's --chown and --chown-recursive are only used on volumes and
caches - the only locations that aren't controlled by the Docker image
itself and thus whose permissions could be "undefined."
Previous commits have taught run-task about the locations of all caches
and volumes. Therefore, we no longer need to manually define paths to
chown. Instead, we can chown as a side-effect of the path being a
cache or a volume.
So, this commit changes run-task to chown caches and volumes
automatically. Since we no longer have a use for --chown and
--chown-recursive, those arguments are removed.
There /could/ be some paths that are caches or volumes but aren't
getting defined as such in Taskgraph. I consider this a bug in
Taskgraph and the recourse is to properly define a path as a cache or
a volume there.
MozReview-Commit-ID: 1yqrhjil6gy
--HG--
extra : rebase_source : 82c9ccb33f09d3ce50a6b7c6021e1e5b77d1b738
See the inline comment for the rationale here.
This check may not catch all volumes and caches. But after subsequent
commits refactor how permissions for caches and volumes are handled,
this edge case will likely result in permissions errors in the task,
so it isn't worth worrying about.
Several Dockerfile have been updated to add missing VOLUME so the check
passes.
In the case of desktop1604-test, we stopped removing
/home/worker/.cache because you can't remove a mount point, which is
what volumes are inside Docker containers.
MozReview-Commit-ID: GEyNkkX00kN
--HG--
extra : rebase_source : 19b3165d8c6e4431aa43dca51828be255e82e34f
We recently changed how image compression works (bug 1350447) and
upgraded to Mercurial 4.1.1 (this bug). It's time to roll a new
image_builder Docker image.
This commit updates the version and hash of the image. The image
is already pushed to Docker Hub/Cloud. We just need to whitelist
it and deploy this change.
MozReview-Commit-ID: KQwJZ2OSGlt
--HG--
extra : rebase_source : 23f2686cd85256178b5d89c452aaeabe4e4198db
Using --no-install-recommends prevents installation of unwanted
packages and helps reduce image bloat. Using it in image_builder
reduces the image size from ~212.5 MB to ~203.6 MB. Not the
biggest savings. But savings nonetheless.
I kept python-pip separate because using --no-install-recommends with
it fails to pull in setuptools, gcc, and other packages required for
building Python packages. Since these packages are only needed
for generating the Docker image, I didn't feel like teasing out
the actual dependencies. It takes a little longer to install
temporary packages and build the image. But the end result should
mostly be the same.
MozReview-Commit-ID: EVURGKFuWzb
--HG--
extra : rebase_source : e1f953079480c6d898569dc7e14dc97890d4f41e
Mike Hommey
Chris AtLee
Gregory Szorc
Bogdan Tara
Dorel Luca
Dustin J. Mitchell
Andreea Pavel
shindli
Simon Fraser
Margareta Eliza Balazs
Tom Prince
Phil Ringnalda