Sometimes xvfb will not start up with the current retry/delay settings. This will
attempt to retry more and delay for longer to ensure xvfb has started up. Common
pieces of this have been factored out into a recipe that all docker images can schare
that need this functionality.
MozReview-Commit-ID: BTXkJkBWLZX
--HG--
extra : rebase_source : 7554d7784768a65deb2c9ccb89586e5f77550e9f
Before, test.sh (duplicated between the desktop-test and
desktop1604-test images) was dropping permissions, creating a workspace,
and executing test-linux.sh. This is functionality now provided by
run-task.
So, convert the test tasks to use run-task.
It's worth noting that creating the workspace directory is no longer
necessary because it is defined as a VOLUME in the Dockerfile or a
cache in some task configurations. However, it is default owned by
root:root, so we do need to instruct run-task to chown it.
Since the test.sh files are no longer used after this change, they have
been deleted. The desktop-test image no longer has any files in the
bin/ directory, so the Dockerfile entry to copy those files has been
removed.
MozReview-Commit-ID: 1BiskrMs6xW
--HG--
extra : rebase_source : 4b16837ad9362844668fe4d6cde5ed07a59a3ccd
extra : source : 8335aa40265b1d17421d06d9e9a180eb8419fe47
It is common for tasks to chown paths before permissions are dropped
from root:root. This commonly occurs when Docker volumes/caches are
involved since they are default owned by root:root and not writable
by any other user.
Since this will likely be a common request, add the functionality to
run-task.
MozReview-Commit-ID: AHmSfY5Ce0S
--HG--
extra : rebase_source : 692177bb730a9d0f99abb6add992c9e3c066d306
Sometimes xvfb will not start up with the current retry/delay settings. This will
attempt to retry more and delay for longer to ensure xvfb has started up. Common
pieces of this have been factored out into a recipe that all docker images can schare
that need this functionality.
MozReview-Commit-ID: 2ww0eT3cIt6
--HG--
extra : rebase_source : f343f8a2bf8370657b63afdd9d739988bbb6261c
Before, test.sh (duplicated between the desktop-test and
desktop1604-test images) was dropping permissions, creating a workspace,
and executing test-linux.sh. This is functionality now provided by
run-task.
So, convert the test tasks to use run-task.
One thing run-task isn't doing is created the workspace. So this
functionality has been moved into test-ubuntu1204.sh and
test-ubuntu1604.sh.
Since the test.sh files are no longer used after this change, they have
been deleted. The desktop-test image no longer has any files in the
bin/ directory, so the Dockerfile entry to copy those files has been
removed.
MozReview-Commit-ID: 1BiskrMs6xW
--HG--
extra : rebase_source : 264efc9b30e62927dece848b9b1eab7542cf2cba
extra : source : 8335aa40265b1d17421d06d9e9a180eb8419fe47
extra : histedit_source : a5ae54a810718d446ded8bab510e6cd605562737
Before, we `curl`d these files at task run time. Why, I'm not sure.
Probably the easiest way to do things before we could add files outside
the directory of the Dockerfile to the image build context. Since that
is no longer a restriction, we just include the file as part of the
built image.
This does mean we rebuild the Docker image if the scripts change. I'm
fine with that.
It's worth noting that both test.sh files look very similar. I
anticipate consolidation in the future...
MozReview-Commit-ID: 48dinmxy6G0
--HG--
extra : rebase_source : 418f11ada7aaf16f3a4c3522f014b42690d44295
extra : histedit_source : df27ba0c3bbff1721d8b278ff65c127e6332cc28
Without this, we potentially mix unicode literals and str and have
implicit type coercion, leading to UnicodeDecodeError or some such.
Oh, the joys of Python.
MozReview-Commit-ID: IeXRjs5Fnj7
--HG--
extra : rebase_source : cba8b13ae845dfaf41a718ba8e60061b8bcffccd
Before, test.sh (duplicated between the desktop-test and
desktop1604-test images) was dropping permissions, creating a workspace,
and executing test-linux.sh. This is functionality now provided by
run-task.
So, convert the test tasks to use run-task.
One thing run-task isn't doing is created the workspace. So this
functionality has been moved into test-ubuntu1204.sh and
test-ubuntu1604.sh.
Since the test.sh files are no longer used after this change, they have
been deleted. The desktop-test image no longer has any files in the
bin/ directory, so the Dockerfile entry to copy those files has been
removed.
MozReview-Commit-ID: 1BiskrMs6xW
--HG--
extra : rebase_source : 75a937321d1850caebbb1eeaab42d04638ef75d9
extra : source : 8335aa40265b1d17421d06d9e9a180eb8419fe47
Before, we `curl`d these files at task run time. Why, I'm not sure.
Probably the easiest way to do things before we could add files outside
the directory of the Dockerfile to the image build context. Since that
is no longer a restriction, we just include the file as part of the
built image.
This does mean we rebuild the Docker image if the scripts change. I'm
fine with that.
It's worth noting that both test.sh files look very similar. I
anticipate consolidation in the future...
MozReview-Commit-ID: 48dinmxy6G0
--HG--
extra : rebase_source : 4d61b2dcb605c7314b872de85db4d46c0199b8df
This effectively reverts the change in 2f0d6ea822b5 (bug 1163057)
to own the files by root:root. Since that change landed, we su
to the "worker" user early during task startup. So there is no more
need to have /home/worker owned by root:root.
MozReview-Commit-ID: 77q9APiDKpb
--HG--
extra : rebase_source : 6f45af9c51fa6c0f9f14ea183b008e38f9388664
The install-mercurial.sh script takes care of installing a global hgrc.
If we want Mercurial settings, that's where they should be defined.
A number of Docker images had redundantly maintained versions of
/home/worker/.hgrc. We've deleted some instances of these hgrc files
because I thought they were cargo culted. It turns out that the config
was added to support one-click loaners (see bug 1250904).
Since there is little harm in enabling extensions globally, we just do
that in the shared install-mercurial.sh script and do away with the
custom hgrc.
It's worth noting that the CA path defined in the custom hgrc is not
needed because the Mercurial package installed by install-mercurial.sh
configures this correctly.
MozReview-Commit-ID: bk4ek99X4z
--HG--
extra : rebase_source : 6ae90f698af05a8814cdb08cf672db470a5a227a
We're giving the same treatment to the Ubuntu 12.04 test image that we
did to the Ubuntu 16.04 image in bug 1293686.
As of this commit, the image can be fully regenerated from changes in
tree: no Docker Hub pushes necessary.
A downside is would could get bit by lack of determinism in image
building. But that's already a problem.
Since the ubuntu1204-test and ubuntu1204-test-upd images are no longer
used, they have been deleted.
MozReview-Commit-ID: 55DwqHOmVM5
--HG--
extra : rebase_source : 222bb9ec096705f1be3f8ab7309a24ac640683f3
In preparation for deleting this image.
MozReview-Commit-ID: Iu9vifGgGE0
--HG--
rename : testing/docker/ubuntu1204-test/system-setup.sh => testing/docker/recipes/ubuntu1204-test-system-setup.sh
extra : rebase_source : a9fb795382766b3d9862b3fd211e97a6281f2c7f
The decision task now uses robustcheckout to get the latest mercurial state.
However, robustcheckout also enforces that the revision it's passed is
actually a revision hash, and not a symbolic name. This will use the --branch
option of robustcheckout if GECKO_HEAD_REF is defined and will use `hg log` to
fill in GECKO_HEAD_REF.
MozReview-Commit-ID: LJikceW4YVg
--HG--
extra : rebase_source : ee9d4d8c472239dbe452e7467e4ebec08a4c5594
extra : source : ad0e20ec7cd249036bdc1ee0638a7499d4a39ed7
The code from the base ubuntu1604-test image has been moved to
the desktop1604-test image. Since ubuntu1604-test is no longer
referenced, we remove it.
With this change, the desktop1604-test Docker image no longer
requires a push to the Docker Hub when updating: it is updated fully
within TaskCluster.
This will result in more image transfers in automation and to
developer machines. But we do gain a lot of convenience updating
the images.
MozReview-Commit-ID: GRpPOfs6pn3
--HG--
extra : rebase_source : 7bac6c7427d0dbee7191a6284fc58f0a2b936b33
In preparation for removing the ubuntu1604-test base image. We also
change the script's location in the image to /setup because that's the
path we use in most images.
MozReview-Commit-ID: CvjTO8hmIkW
--HG--
rename : testing/docker/ubuntu1604-test/system-setup.sh => testing/docker/recipes/ubuntu1604-test-system-setup.sh
extra : rebase_source : 77b6b0028cb9d054b37584c314cfb7eb1d8b9a59
Pinning these packages breaks Ubuntu package management because
other packages we install explicitly require a newer version.
Attempting to pin these results in packages getting removed or refusal
to install.
MozReview-Commit-ID: AhQbjavCKI1
--HG--
extra : rebase_source : 1774eaf447a46d7244cb934b0e5671b140b923fa
As bug 1291940 shows, aufs can be a real performance drag.
Most performance critical paths in containers are caches. This allows
the use of the host filesystem (instead of aufs), which is much faster
under load.
Unfortunately, most caches are stripped on Try. This means paths that
would be served from the fast host filesystem are now being handled
by aufs. This can make Try jobs slower than non-Try jobs.
This commit declares Docker volumes for popular cache paths. If the
cache is present, the cache takes precedence over the declared volume.
If the cache isn't present, Docker falls back to an empty volume that's
used for the task and only the task.
It's worth noting that desktop-build is updated redundantly with
centos6-build. We currently can't rebuild centos6-build because of
a determinism issue (bug 1272629) breaking valgrind. So, I work around
this by declaring the volume in the top-most "desktop-build" layer.
This hack can be removed once we merge the images.
MozReview-Commit-ID: 7JtxAfI7ED5
--HG--
extra : rebase_source : aec1675c3393d767f5d10130553148155190f19d
This was likely a cargo cult. Mercurial's config is now managed by
install-mercurial.sh, which is called by the base image.
MozReview-Commit-ID: 3uNN880cm2f
--HG--
extra : rebase_source : c7555ca8ab41356ab10f4c02938a9c8add1acc61
And make it deterministic since we are using a vendored file.
MozReview-Commit-ID: 4ZvMRt9BSZi
--HG--
extra : rebase_source : e2dbdb2d83bc910592eddd598e7c51748e8be4e2
Without this, we'll use the latest Ubuntu release at image build time!
MozReview-Commit-ID: DmUcwKPkL78
--HG--
extra : rebase_source : 45edb72205bba3667abdf5d9ac79f1f66811ddc5
This also bumps the Mercurial version from 3.7.3 to 3.8.4.
MozReview-Commit-ID: 2qfDLTFqlaj
--HG--
extra : rebase_source : f208a0df7a47cec083e873934c8140c048c46c06
More determinism. More security. More reliable. Vendoring.
The path to the installed tooltool.py has also been changed to /setup
for consistency with other images.
MozReview-Commit-ID: JnUS35AyPus
--HG--
extra : rebase_source : d3dbc5119d0d95e28e3a6c789572a9dbf0800e9f
I produced Mercurial .deb packages for Ubuntu 12.04 and uploaded them to
tooltool. I have updated the Mercurial installation script to detect
Ubuntu 12.04 and install the appropriate Mercurial packages.
MozReview-Commit-ID: GsNFOhhms7b
--HG--
extra : rebase_source : d64016382035ae99a3be9abc5bf974ab710c058b
Currently, install-mercurial.sh only supports Ubuntu 16.04. In
preparation for supporting other distros, we add code for performing
OS/distro detection.
MozReview-Commit-ID: 9a6pp1Qprcr
--HG--
extra : rebase_source : 2b1c892ddb6ed4252d730a4ba08edb7a158fb3ec
Without this pinning, the image doesn't build deterministically and
fails the version checks that were previously in this file. At the
time this was written, the libgl1-mesa packages were on version
11.2.0-1ubuntu2.1 upstream and building the Docker image locally
failed because these versions would get installed.
A risk to pinning is we'll drift out of date with the OS we're testing
and won't detect failures on packages users in the wild are running.
But on the flip side, automation will be able to build the image
reliably and there won't be "random" failures due to automation
picking up new package versions.
MozReview-Commit-ID: 5MSEbsiNjF1
--HG--
extra : rebase_source : 14b57ee441cb12ce866f402ca94d270c86d1cc4d
Before, we were returning None, which gets converted to 0.
Derp.
Also fix a flake8 failure introduced by 9f5fbb3066c9.
We'll also need to generate a new decision image. But that will
require someone with TC privileges to be around. That can be done
in a separate commit to unblock this from landing and fixing
consumers of run-task that aren't the decision image.
MozReview-Commit-ID: 6XuoIxjDozF
--HG--
extra : amend_source : 4a3047d09d9bdce5a173a71ec3c68b8c4579a910
Before, we were returning None, which gets converted to 0.
Derp.
Also fix a flake8 failure introduced by 9f5fbb3066c9.
We'll also need to generate a new decision image. But that will
require someone with TC privileges to be around. That can be done
in a separate commit to unblock this from landing and fixing
consumers of run-task that aren't the decision image.
MozReview-Commit-ID: 6XuoIxjDozF
--HG--
extra : rebase_source : 836f4996cb9be24e19504f73792b159f0ffbdbee
This commit does a lot. But it's really not too difficult to comprehend
once you focus on the final state, which is basically the same as the
"lint" image and derived tasks.
Before, the "decision" image contained a "checkout-gecko" script and
"run-action" and "run-decision" scripts. The latter 2 invoked the first
script.
The "run-task" script basically does what the combination of these
scripts were doing before. So we switch to it.
While we're here, we also replaced the custom Mercurial installation in
this image with the shared install-mercurial.sh script. The
system-setup.sh script for the decision image is now short and sweet.
The YAML files for tasks using this image have been updated to use
"run-task." We no longer have to pass an environment variable to hold
command arguments. So we revert to putting these arguments inline in
the task's command. Dustin never liked passing the arguments as
environment variables, so it should make him happy ;) We add
--log-no-times because "run-task" prefixes its own timestamps on output
lines.
The path to the Gecko checkout has been changed from
/home/worker/workspace to /home/worker/checkouts to match changes made
in bug 1289643.
Finally, since "checkout-gecko-and-run" is no longer used, we delete it.
The Docker image version has been bumped accordingly.
MozReview-Commit-ID: HAGuCz6668Q
--HG--
extra : rebase_source : 9586873c21d67f8ce869b7ceee221e8fa8775539
Before, we simply executed scripts inside Docker containers. This
frequently resulted in a wall of text with command output. It was
difficult to discern things like the time spent performing certain
actions.
Before, individual tasks had to drop permissions from the default
root user themselves. Dropping permissions isn't exactly a trivial
thing to do and a number of tasks didn't do it or did it wrong.
Before, we had a "checkout-gecko-and-run" script that kinda/sorta
did common activities for us. But it was written as a shell script
and doing advanced things was difficult.
This commit can be treated as a rewrite of "checkout-gecko-and-run"
as a Python script. But it also does a bit more. It prefixes output
with timestamps so we know how long operations took. It features more
robust argument parsing, so we can add new features more easily.
To prove the new wrapper script works, the lint image and all tasks
using it have been converted to use it.
MozReview-Commit-ID: 5d95u5Xebtq
--HG--
extra : rebase_source : 3a1d84782b01d7743e846bd0c04d7867813dd8a3
The reason tooltool.py was vendored in testing/docker/decision was
because locally built Docker images were using vanilla `docker build`
and didn't know about our special Dockerfile syntax to allow the
inclusion of images from outside the directory where the Dockerfile
was located.
Now that locally-built Docker images know of our special Dockerfile
syntax, we can include files from anywhere. So, move tooltool.py
to a shared directory, away from the decision image.
I didn't bump the version of the decision image because there are
a few more things I want to do to this image, such as have it use
the `checkout-gecko-and-run` script instead of its own script.
I think I'll do that in a separate bug, however.
MozReview-Commit-ID: 12ze9GJFMJq
--HG--
rename : testing/docker/decision/tooltool.py => testing/docker/recipes/tooltool.py
extra : rebase_source : cfc2fa5759a3ec708b6e1736e9a2da03f0d25428
build.sh had been reduced to invoking `docker`. We move that invocation
to Python and remove build.sh. Long live build.sh!
MozReview-Commit-ID: FQBDJv4HSaU
--HG--
extra : rebase_source : 82979f2526ab1ab6dd5c25032b8d81074c797d94
Now that tester-device is gone, there are no more images using custom
build.sh scripts and that feature can be deleted. Yay simplicity.
MozReview-Commit-ID: 5JqFZ2HOsWs
--HG--
extra : rebase_source : 437bc8e877be065bf85db81422f3c1ddd5ac2800
There are no references to tester-device in tree or in the gaia repo.
Since it appears to be unused, remove it.
According to garndt, this was used for testing in the remote device
lab, which he things is no longer happening.
The reason I found this is because it is the only thing using a custom
"build.sh" to create Docker images. I'm rewriting the Docker image
building functionality and tester-device is a one-off interfering
with that work. Making it go away is the easiest way to unblock me.
MozReview-Commit-ID: 4veem4uIXXi
--HG--
extra : rebase_source : c2b5e59bdf5e84e7d7132c04bdf5b46c3392b37a
extra : amend_source : ff20838a8c828451412caf57c89efc59b2db202e
We already had code for resolving the image registry and tag. We
refactored it slightly to be more useful then changed build.sh to
accept the tag as an argument.
At this point, build.sh is basically a wrapper around `docker`. But
there's a special case for executing custom "build.sh" files we
need to eliminate first...
MozReview-Commit-ID: A9HVvxgCdG2
--HG--
extra : rebase_source : 30a408860aea619813f32723fe960d1224b5dbc7
Gregory Szorc
Gregory Arndt
Sebastian Kaspari
Wes Kocher
Alexandre Lissy
Anthony Miyaguchi
Dustin J. Mitchell
Iris Hsiao