332a08725ed0 changed the store path logic in a quick and crude manner.
The code could lead to multiple shared stores existing on a cache
if checkouts were in different parent directories.
This commit refactors the code to explicitly declare a path to the
shared hg store. This restores the behavior of ensuring there is only
a single shared store per cache.
MozReview-Commit-ID: 19Aa1QVrVQ8
--HG--
extra : rebase_source : f74c0ab9cba2bf18e335f24c86dd095335961723
This definitely isn't all of it. But I got the obvious chunks. Still
need to clean up mozharness.
MozReview-Commit-ID: JTZBydP3i2r
--HG--
extra : rebase_source : 782401359036751896ef6c153a81a06cb14031bb
We were seeing issues with the Mercurial working directory not being
pristine. While I can't reproduce this, I have a hunch it is due to
mixing and matching stores and checkouts in TaskCluster. For example,
if a worker supports running concurrent tasks and 2 tasks arrive at
the same time, the caches for the store and checkout may look like:
(store0, checkout0)
(store1, checkout1)
However, the next task may get:
(store1, checkout0)
This may confuse Mercurial.
This commit eliminates the "hg-shared" cache and places the shared
stores as a sibling directory of the checkout.
MozReview-Commit-ID: 8SzyS6wWf9C
--HG--
extra : rebase_source : 6205f3fe944a6ad2cb833a5a7c0ae5ba136eaea6
The working directory state in automation is somehow incorrect. To
help debug what is doing on, we increase the verbosity of the
`hg robustcheckout` command and perform an explicit `hg status`
after `hg robustcheckout` and verify that the working directory has
no differences from the pristine checkout state.
`hg status` can be expensive. So this is meant as a temporary
stop-gap until we can figure out the underlying problem.
MozReview-Commit-ID: LIHmoMlxNsG
--HG--
extra : rebase_source : 358b403c16868a83092a1fa926d7e2a669b4620d
Instead of having the desktop[1604]-test images currently bake in a
minidump_stackwalk binary from some random S3 URL, use the binary listed
in the in-tree tooltool manifest that we use for buildbot test jobs. As
a nice side-effect, this will ensure that the desktop-test images get
rebuilt whenever that manifest is updated with a new version, so they
will continue to use the right version in the future.
MozReview-Commit-ID: 6bThddwq6p1
--HG--
extra : rebase_source : 31111e1ae8c10a72c3635bc365babe7d5b1fb4e3
Instead of having the desktop[1604]-test images currently bake in a
minidump_stackwalk binary from some random S3 URL, use the binary listed
in the in-tree tooltool manifest that we use for buildbot test jobs. As
a nice side-effect, this will ensure that the desktop-test images get
rebuilt whenever that manifest is updated with a new version, so they
will continue to use the right version in the future.
MozReview-Commit-ID: 6bThddwq6p1
--HG--
extra : rebase_source : eafefb980fe398dda435ad066145ffb8d7003d8f
run-task is our new universal wrapper for executing tasks. Add it
to desktop-build.
MozReview-Commit-ID: BCYHVRdUopQ
--HG--
extra : rebase_source : e2106b76c222a410920086faecad4d45833ff73c
In preparation for switching desktop-build to use run-task and
its VCS management.
MozReview-Commit-ID: 17WBMQhJxaV
--HG--
extra : rebase_source : a79709f5164687782b540c356c3888ee4e298fbc
As part of this, we had to teach install-mercurial.sh to detect
CentOS and install from RPM or source. While we can support
installing from an RPM on CentOS 6, this code is currently disabled
because the RPM we have is built against Python 2.6, which doesn't
support TLS 1.2. Since we have Python 2.7 on the image and this
Python 2.7 install supports TLS 1.2, we build Mercurial from source
using this Python 2.7 install.
We also added a "system setup" shell script. This matches the
conventions used for the desktop-test images.
MozReview-Commit-ID: 7cHN54n7aQF
--HG--
extra : rebase_source : b8ccddf0da76e94ac4064786f3b0ffa0719c9078
extra : source : cc6b486fb6bf92a408fb608283ecd8de4c4419c2
I will add more stuff to common.sh in the future in order to justify
its existence. Not in this bug though.
MozReview-Commit-ID: Lx7MJwBMH0w
--HG--
extra : rebase_source : fe6f51a9b6910abd9dedfda54c0bc8ebd3c3551e
Vendoring: more reliable, more determinism, more secure.
MozReview-Commit-ID: BYUUj4ZpndD
--HG--
extra : rebase_source : 4a1125efcca1fb50d5c191d3aadb0d11d442b628
Build tasks currently require a checkout of the build/tools repository.
I wish this weren't true and that all files references from this repo
were part of mozilla-central or tooltool, but that's how things are.
In preparation for running build tasks with run-task, teach run-task
to perform a checkout of the build/tools repo. Ideally we'd support
configuring the URL to this repository. But I'm not implementing that
since I'd prefer we stop relying on the build/tools repo.
MozReview-Commit-ID: B2Y1NwS3niO
--HG--
extra : rebase_source : bcad2f101e94411a5defd655247ed4ace250a852
Previously, we assumed we only could have a single version control
checkout: Gecko/Firefox. The code reflected this by not passing
arguments to the vcs_checkout function.
Upcoming commits will introduce the need to perform a checkout of the
build/tools repository. In preparation for this, refactor the VCS
functionality so it is generic and can work on any repo.
MozReview-Commit-ID: B0Act9fz2Ee
--HG--
extra : rebase_source : 512d0ccb15adce5ed95c4623562eb47535aef29b
os.walk() won't explicitly yield the root directory. So we need to
update it explicitly when doing a recursive chown.
MozReview-Commit-ID: JC0PNsk5gFK
--HG--
extra : rebase_source : ddfd437cd5e6bffb8780baf23813b88dd06e471d
When we initially implemented support for robustcheckout, we didn't
have the magic "%include" syntax in Dockerfiles. So we used tooltool
to download robustcheckout.py to the image.
Now that we have nice things, we can use the vendored robustcheckout.py
file.
As part of this, I realized we're inconsistently using /tmp, /setup
and /build for files used during image building. That should probably
be cleaned up. I'd rather not bloat scope for this bug, however.
MozReview-Commit-ID: D99Gcdw1DId
--HG--
extra : rebase_source : 3f361da3088988423b50786d990afba662d297d2
Python processes with a TTY have stdout line buffered by default.
Python processes without a TTY have buffered output.
Mercurial inherits whatever Python's output buffering behavior is.
This means if we invoke Mercurial without a TTY, stdout and stderr
will be fully buffered. This means output may not be sent until
there is enough output to flush the output buffer.
A consequence of this is that timings reported for `hg` commands
invoked by run-task are inaccurate. In addition, output order is
incorrect. This is because Mercurial's progress indicators print
to stderr and flush when written. This means stderr output is
getting seen by run-task but stdout remains buffered.
This commit forces Python/Mercurial to not buffer stderr and stdout
by setting the PYTHONUNBUFFERED=1 environment variable.
MozReview-Commit-ID: 7lMdrjRMBqz
--HG--
extra : rebase_source : 198ced0053fe6071a45c9df8b044b24983c225cc
The Python code is now intelligent enough to add this flag on the
command line if supported. Eliminate the copy pasta and help prevent
cargo culting.
MozReview-Commit-ID: H4rbjbbgtRd
--HG--
extra : rebase_source : 2f3acf666d73260ed97ad877f365b3091186f9e2
install-mercurial.sh was switching directories to /usr/local/mercurial
resulting in following actions adding files to that path. We don't
want that. So avoid the `cd` in install-mercurial.sh.
The main side effect of this change is that the desktop-test image is
now ~1.2 GB smaller because files aren't being saved to
/usr/local/mercurial.
MozReview-Commit-ID: Kyv8oXtvsda
--HG--
extra : rebase_source : f73dc49bf0fe457aebc6c858cd3fcaf6ce59ce6d
The only difference in these files was the order that pulseaudio is
started and whether compiz is started. We rename test-ubuntu1604.sh
to test-ubuntu.sh, add some distro release detection, and add
some conditional branches so it works on both Ubuntu 12.04 and 16.04.
MozReview-Commit-ID: CaSfuDxss3d
--HG--
rename : taskcluster/scripts/tester/test-ubuntu1604.sh => taskcluster/scripts/tester/test-ubuntu.sh
extra : rebase_source : 29040c1cf7baedda0aaeff4bd788d5d400c127f1
extra : source : f227ea4d52fed84e3e682de0aa4cb8869539d645
We can't add it to the base image because rebuilding the base image
breaks Valgrind due to non-deterministic package version installation
(read the bug for the ugly backstory).
MozReview-Commit-ID: ARKJZfNCRFc
--HG--
extra : rebase_source : 6a36b1289d121367c89c986f86faf1e34a38a906
extra : source : 66c7af8b26544e79b39b7180cb7338bbc2642064
The only difference in these files was the order that pulseaudio is
started and whether compiz is started. We rename test-ubuntu1604.sh
to test-ubuntu.sh, add some distro release detection, and add
some conditional branches so it works on both Ubuntu 12.04 and 16.04.
MozReview-Commit-ID: CaSfuDxss3d
--HG--
rename : taskcluster/scripts/tester/test-ubuntu1604.sh => taskcluster/scripts/tester/test-ubuntu.sh
extra : rebase_source : 2153d24fbf8208851a6df8728b8a820166278751
Previously, when recursively changing ownership on directories we would
only change the owner. We saw some permission denied failures in
automation where the new owner couldn't modify files or directories.
This *might* be due to the owner write bits not always being set. Or
it could be something else (such as a filesystem bug - *cough* AUFS
*cough*).
This commit changes our recursive chown implementation to ensure owner
read, write, and execute bits are set on directories.
Because we're now always calling stat(), the code for calling chown()
is made conditional because we have the stat information and can avoid
the extra system call if it would be a no-op.
MozReview-Commit-ID: JT9q3QR4Sit
--HG--
extra : rebase_source : 86ceecb3a9381390670d47d46862dad4fc38c403
This commit does a few things. First, it introduces a property on the
"test_description" schema that, if defined, will cause run-task to
perform a gecko checkout. The presence of the property also configures
the needed scopes and caches.
Second, we introduce the property on web platform test tasks so a
Gecko checkout is present. We also add volumes for the Mercurial
paths to the Docker images. We strictly only need this for
desktop1604-test since WPT tests don't run on desktop-test. However,
desktop-test and desktop1604-test are nearly mirror images of each
other and I feel it is best to keep them in sync.
This commit will make WPT tasks slower on average because they will
need to create a checkout. To add salt to the wound, the checkout
isn't used. However, we need to prove that performing checkouts in
test tasks in automation works at scale. I'd prefer to have this running
for a few weeks and incurring a wall time execution penalty than to
have a giant series of commits backed out because source checkouts
aren't working.
MozReview-Commit-ID: 9UrSWSSmr3w
--HG--
extra : rebase_source : 7b3786f5c612d47dc3b0e165b4abe0c47e8af9ed
Before, test.sh (duplicated between the desktop-test and
desktop1604-test images) was dropping permissions, creating a workspace,
and executing test-linux.sh. This is functionality now provided by
run-task.
So, convert the test tasks to use run-task.
It's worth noting that creating the workspace directory is no longer
necessary because it is defined as a VOLUME in the Dockerfile or a
cache in some task configurations. However, it is default owned by
root:root, so we do need to instruct run-task to chown it.
Since the test.sh files are no longer used after this change, they have
been deleted. The desktop-test image no longer has any files in the
bin/ directory, so the Dockerfile entry to copy those files has been
removed.
MozReview-Commit-ID: 1BiskrMs6xW
--HG--
extra : rebase_source : f4b8f320fafa2da1aa4b06c9cbef76c69acdae8f
extra : intermediate-source : 905fb4a53386c76d931e4a04a31d03739a00a1ff
extra : source : 8335aa40265b1d17421d06d9e9a180eb8419fe47
It is common for tasks to chown paths before permissions are dropped
from root:root. This commonly occurs when Docker volumes/caches are
involved since they are default owned by root:root and not writable
by any other user.
Since this will likely be a common request, add the functionality to
run-task.
MozReview-Commit-ID: AHmSfY5Ce0S
--HG--
extra : rebase_source : 705891869413495b015b490ee75efd40b26d24d8
extra : source : 213a4986ccfd71f04f822e56b1a9ad9505ed5e98
We just upgraded our run-time environment to Mercurial 3.9.
3.9 features a new [hostsecurity] config section and allows certificate
fingerprints to be defined using SHA-256 hashes (not just SHA-1).
A TaskCluster secret with the Mercurial 3.9 fingerprint format has
been added. This commit takes advantage of it.
MozReview-Commit-ID: 5NwJl9zOse2
--HG--
extra : source : ba338ab6c809066651def6e1a611e7f1971668fe
extra : intermediate-source : d31f5e1edb53ced0cfd71d6692354713e40615be
extra : histedit_source : 9281ab19a20248077fa949562bd96a62c69431de
We just upgraded our run-time environment to Mercurial 3.9.
3.9 features a new [hostsecurity] config section and allows certificate
fingerprints to be defined using SHA-256 hashes (not just SHA-1).
A TaskCluster secret with the Mercurial 3.9 fingerprint format has
been added. This commit takes advantage of it.
MozReview-Commit-ID: 5NwJl9zOse2
--HG--
extra : rebase_source : dc6d342f9e964a59ce2181f95f25d40155571fec
extra : source : ba338ab6c809066651def6e1a611e7f1971668fe
Containing the Mercurial 3.9.1 version bump and the change to pin the
hg.mo fingerprint from a TC secret.
MozReview-Commit-ID: LVU7P0LqIvD
--HG--
extra : rebase_source : 8ad46e014ba9840c9972b51ea43e8ccf14492cf2
We just upgraded our run-time environment to Mercurial 3.9.
3.9 features a new [hostsecurity] config section and allows certificate
fingerprints to be defined using SHA-256 hashes (not just SHA-1).
A TaskCluster secret with the Mercurial 3.9 fingerprint format has
been added. This commit takes advantage of it.
MozReview-Commit-ID: 5NwJl9zOse2
--HG--
extra : rebase_source : fc985712d299242aec8f8fbc328eba8baca95508
curl and jq were previously used to fetch and parse the TC secret.
We now use Python for that. So remove the unused packages.
This reduces the Docker image size by ~10MB.
MozReview-Commit-ID: Nl7fC1aG7w
--HG--
extra : rebase_source : e1bfe76f0e2efd06c2e2e8a94f6c3fb3dc4c0d50
Mercurial 3.9 changes the default security settings to make Mercurial
secure by default. It is important for Firefox's automation to be
secure.
MozReview-Commit-ID: IF7Z74111hI
--HG--
extra : rebase_source : 3bc3cbe7688a8ecdd2593cbe202e456a649b49e1
When I wrote this initially, I was told it was not possible to inherit
from desktop-build. That may have changed, but I still don't know how
to inherit, so we duplicate the logic here, inviting bugs of this
class. The underlying logic moves slowly so this is an acceptable
approach.
MozReview-Commit-ID: AsWdbp0QK5K
--HG--
rename : testing/docker/desktop-build/dot-config/pip/pip.conf => testing/docker/android-gradle-build/dot-config/pip/pip.conf
extra : rebase_source : b13d97f5faabfaa42d1068b216d2c5ad5052154c
Sometimes xvfb will not start up with the current retry/delay settings. This will
attempt to retry more and delay for longer to ensure xvfb has started up. Common
pieces of this have been factored out into a recipe that all docker images can schare
that need this functionality.
MozReview-Commit-ID: BTXkJkBWLZX
--HG--
extra : rebase_source : 7554d7784768a65deb2c9ccb89586e5f77550e9f
Gregory Szorc
Wes Kocher
CuriousLearner
Justin Wood
Ted Mielczarek
Grigory Kruglov
Carsten "Tomcat" Book
Al Tsai
Nick Alexander
Gregory Arndt
Sebastian Kaspari