We now have access to workers running on EC2 instances with dozens of
vCPUs. gecko-<L>-b-linux-large is m4.10xlarge, m5.12xlarge, c5.9xlarge,
or c4.8xlarge. gecko-<L>-b-linux-xlarge is m5.24xlarge, m4.16xlarge,
or c5.18xlarge.
Experimentation reveals that Clang tasks are the only tasks that
are CPU efficient enough (read: cost effective) to run on these
larger worker types.
This commit defines the new worker types and switches Clang toolchain
tasks to run on the new workers. clang5 and clang6 tasks take ~30 minutes
on the -large variant but ~17 minutes on the -xlarge variant. All other
tasks don't show as linear of a speedup. So running them on the
-xlarge variant isn't justified.
As part of this change, Mac toolchain tasks have been converted
to run on gecko-<L>-b-linux* workers. The gecko-<L>-b-macosx64 workers
are actually Linux. IMO the b-macosx64 worker type is no longer needed.
Moving the toolchain tasks off the worker should hopefully not be very
controversial.
MozReview-Commit-ID: HynQPMWiWHo
--HG--
extra : rebase_source : 1142767e2a51c17880909ec6f15b694db8a43af2
Since comm-central doesn't have integration branches like autoland, we want to
always disable sccache on nightly builds, rather than being able to key off of
the tree to determine whether to enable it. This is currently done by not
including the sccache toolchain. However, the mozconfig files expect sccache to be
available if `SCCACHE_DISABLE` isn't set, which wasn't the case on windows.
Differential Revision: https://phabricator.services.mozilla.com/D603
--HG--
extra : rebase_source : 565d0b6a3b24a23f6ae844a8cd97b9824e9f5282
extra : amend_source : 8c1679a71a2baf04a0dc83074234c5101705b806
Also add rc-{google-play-track,rollout-percentage} for RC pushapk.
One nice side effect of using the same push-apk kind: we don't re-run push-apk during the ship_fennec relpro flavor if we've run the ship_fennec_rc flavor with the same build. (Google Play would reject the same buildid.)
This is really for bug 1433536, but MozReview is forcing me to include this patch with the others for reasons.
MozReview-Commit-ID: 69ymqVLv9E2
--HG--
extra : rebase_source : b87d4dd2394788a5452ff3f52a8ca5022a15b9ee
extra : intermediate-source : 7a826d274a4828018a836cf1149df29d403a7c11
extra : source : a3c60b0370a3e08ce765f87c1d7d5dad24879881
This used to only be relevant to Devedition and Firefox releases.
In bug 1433536 we're going to add RC Fennec releases. Let's rename
the parameter now, for less parameters churn.
MozReview-Commit-ID: 28e1Y5FG4On
--HG--
extra : rebase_source : 59d41887d856481ab85bb8d2221dfcebca4430b0
extra : source : 4c96e312d88a3f7037c1eb39a3031d4baf997015
also clean up and move more config to the promotion config.
MozReview-Commit-ID: FmTWNNPcEaZ
--HG--
extra : rebase_source : 40431217fafb6796dbd65c7dfeab0e891ac1bbd4
extra : source : 0f5418a83477c1b6b221e4d28515792410e504d0
This is a new issue that gets linted with flake8 3.5.0. Basically you should
never use a blank except: statement.
This will catch all exceptions, including KeyboardInterrupt and SystemExit
(which is likely not intended). If a catch all is needed, use
`except: Exception`. If you *really* mean to also catch KeyboardInterrupt et
al, use `except: BaseException`.
Of course, being specific is often better than a catch all.
MozReview-Commit-ID: FKx80MLO4RN
--HG--
extra : rebase_source : 7c74a7d0d81f2c984b47aff3a0ee3448b791177b
This is basically the same deal as e331a3b9fae2. Caching is hard.
MozReview-Commit-ID: 9uWHHdnHgq1
--HG--
extra : rebase_source : fc7107a018f067d83dbee3796d8861c5187823b2
The apt in Debian stretch doesn't allow repositories with a Release file
not being GPG signed. Setting up GPG signatures on the
taskcluster-artifact-based repositories is a tricky process, and not
strictly necessary. It turns out not creating a Release file at all
works just as well, and works across all current Debian versions. The
packages priorities remain the same, such that packages from those
repositories are still prefered over the ones from the main Debian
repository (as long as versions are higher).
See comment in cloud-mirror-workaround.sh for that part.
--HG--
extra : rebase_source : df5af330859a314285a6c1922d899489997d2f19
So far, the best we've been able to do is to upload an image to the
docker hub, and point an image's Dockerfile's FROM to the version
uploaded onto the hub.
That is a cumbersome process, and makes the use of "layered" docker
images painful.
This change allows to declare a parent docker image in the
taskcluster/ci/docker-image/kind.yml definitions, which will be
automatically loaded before building the image. The Dockerfile can then
reference the image, using the DOCKER_IMAGE_PARENT argument, which will
contain the full image name:tag.
Some details are left off, for now, such as VOLUMEs. At this point,
VOLUMEs should all be defined in leaf docker images.
--HG--
extra : rebase_source : 221cff0ca5a91d694ff5c3626fe707c15ba45e23
Now that `mach taskcluster-build-image` can, we can avoid all the manual
handling based on curl and jq in the image builder.
An additional advantage on relying on `mach taskcluster-build-image`
doing more is that less changes to the build-image.sh script will be
necessary, and thus less updates of the image builder docker image.
--HG--
extra : rebase_source : dd174d60675e41e4391894f28235c674c1840829
This allows to avoid writing out a tar file to then extract it to feed
it to `docker build`. This is essentially what the image-builder docker
image does, except it uses a temporary file for the tar.
--HG--
extra : rebase_source : 8275d737e02714fc198d3ba3d3e62e3f18d8e0bf
While spawning `docker load` is likely to work on developer machines,
on automation, it requires a docker client that is the exact same
version as the server running on the taskcluster worker for
docker-in-docker, which is not convenient. The API required for `docker
load` is rather simple, though, and can be mimicked quite easily.
While this change in itself is not necessary for developer machines,
it will allow to re-use the same command for the image-builder to
load a parent docker images when deriving one from another. We could
keep a code branch using `docker load` but it seems wasteful to maintain
two branches when one can work for both use cases.
--HG--
extra : rebase_source : d72956d7dd329b92564cbaa3fbfe0687d4d5d994
The zstd command we spawn, if available at all, might be the wrong
version: zstd changed its stream format in an incompatible way at some
point, and the version shipped in e.g. Ubuntu 16.04 uses the old format,
while the version taskcluster relies on uses the new format.
Relying on gps's zstandard library allows to ensure we use the right
version. Another advantage is that we can trivially pip install it in a
virtualenv if it isn't available on the system running the command.
If we're ridding ourselves of the subprocess spawning for zstd, we might
as well cover curl as well. Especially considering the error handling
when subprocesses are involved is not trivial, such that the current
error handling code is actually broken and leads to dead-lock
conditions, when, for example, curl is still waiting for the python side
to read data, but the python side is not reading data anymore because
an exception was thrown in the tar reading loop.
--HG--
extra : rebase_source : 054c37cfaa68bf475b37545ebaa99144584b93d4
The used pattern:
except Exception:
error = sys.exc_info()[0]
finally:
...
if error:
raise error
actually loses everything that is interesting about the original
exception. Not catching the exception just makes it thrown up the stack,
except when a different exception is thrown from the finally block,
which is what that if error: raise error is attempting to do... except
it doesn't throw the original exception, but its type only.
--HG--
extra : rebase_source : 17601fcc90fcdfefd93c4267f3cd33425d5326fd
Now that we don't need to read the contents of a file to hash the
contents of a docker image context, we can avoid creating a file
in generate_context_hash.
--HG--
extra : rebase_source : 98abe9bfdc48b612a3d251296991d0f769b449fd
This will allow us, down the line, to avoid creating a file at all in
some cases.
--HG--
extra : rebase_source : e4ea40341836cf24aa6d61c905b2efa660ee13f2
Gregory Szorc
Tom Prince
Gurzau Raul
Narcis Beleuzu
Rail Aliiev
Marco Castelluccio
Nick Thomas
Simon Fraser
Brian Stack
Ben Hearsum
Aki Sasaki
Andrew Halberstadt
Geoff Brown
Johan Lorenzo
Cosmin Sabou
shindli
Joel Maher
Mike Hommey