when an instance is terminated while it is still running a task, the generic worker process exits with an interrupt exit code. this change treats such exit codes as an exception which triggers a task retry
Differential Revision: https://phabricator.services.mozilla.com/D29417
--HG--
extra : moz-landing-system : lando
This allows us to fix a regression where -sw tasks were scheduled on autoland/inbound.
Differential Revision: https://phabricator.services.mozilla.com/D29259
--HG--
extra : moz-landing-system : lando
Some groups of tasks need to share the same profile data. For example,
Android PGO builds and Android Nightly builds both use the
generate-profile-android-api-16/pgo task for profile data. Previously
this was done with a text substitution, but this is a bit hacky and
doesn't easily scale with different build types.
Allowing use_pgo to be a string means we can just directly point to the
generate-profile task that contains the profile data to be used in a PGO
build.
Differential Revision: https://phabricator.services.mozilla.com/D29247
--HG--
extra : moz-landing-system : lando
Bug 1474897 changed things such that Windows builds ended up in the
linux/macosx branch. That still works somehow, but ends up breaking when
wrapping with run-task. This change restores the originally intended
command line.
Differential Revision: https://phabricator.services.mozilla.com/D28017
--HG--
extra : moz-landing-system : lando
Currently we have the concept of a "suite" and a "flavour" in our task
configuration. Typically, the "suite" refers to the high-level test harness
like "mochitest" or "reftest", whereas the flavour is more specific, e.g
"browser-chrome-instrumentation" or "crashtest". However the line between suite
and flavour is not applied with any semblance of consistency which results in
inconsistent naming throughout the tree.
This patch gets rid of the concept of "flavours" entirely (at least when it
comes to task configuration). A suite is a type of test run, for example:
- mochitest-plain
- mochitest-devtools-chrome
- mochitest-browser-chrome-instrumentation
- jsreftest
- reftest
- firefox-ui-functional-remote
etc
There is no confusion here between suites and flavours because flavours don't
exist. However, there are a couple of places where we *do* need to know what
"test harness" is used to run a suite. These cases are:
1. For SCHEDULES moz.build rules
2. For the desktop_unittest.py mozharness script which takes arguments like
--mochitest-suite=browser (this is not a compelling use of this information
and should be refactored to work more like the android_emulator_unittest.py
script)
So to get this information, this patch introduces a new concept of a "category"
which is the overall "test harness" that runs the suite. For many suites, the
"category" is identical to the suite name. Unlike flavours, "categories" have
no bearing on how we call or refer to the suite.
Differential Revision: https://phabricator.services.mozilla.com/D27554
--HG--
extra : moz-landing-system : lando
We are starting to spin off more and more "variants" of test suites. These are
usually just duplicates of our pre-existing tasks, except with an additional
pref set.
Currently there are two variants (serviceworker-e10s and socketprocess-e10s),
but a third will be added soon (fission). This change ensures we handle these
types of requests in a consistent and well defined manner. It also splits tasks
in a loop, so we don't accidentally risk combinatorial explosion.
Variants should typically be reserved for very large changes that will impact
the entire codebase (think e10s).
Differential Revision: https://phabricator.services.mozilla.com/D28061
--HG--
extra : moz-landing-system : lando
Many tasks (release tasks and cached tasks, in particular) should be re-run rather
than retriggered. Disable retrigger action for those tasks by default.
Differential Revision: https://phabricator.services.mozilla.com/D27206
--HG--
extra : moz-landing-system : lando
Since e10s is the default configuration, we shouldn't explicitly mark things
with the "-e10s" suffix. Instead we should mark things that *don't* run with
'e10s. This patch removes '-e10s' from all treeherder group symbols and task
labels, adds the "-1proc" suffix to tasks that are non-e10s.
Differential Revision: https://phabricator.services.mozilla.com/D25958
--HG--
extra : moz-landing-system : lando
Much of this was already reviewed in D21473 (my test change where I developed the payload modifications and that pointed tests at my test queue).
This change keeps the payload changes from D21473, but points at the new 'real' queues we'll be using.
Differential Revision: https://phabricator.services.mozilla.com/D25009
--HG--
extra : moz-landing-system : lando
So, instead of fetches['by-test-platform']['fetch'], we have
fetches['fetch']['by-test-platform'].
Differential Revision: https://phabricator.services.mozilla.com/D27227
--HG--
extra : moz-landing-system : lando
Now that we have added the necessary scopes to `ci-configuration`,
we can add the in-tree scopes to give tasks access to the
`hgmointernal` config Taskcluster secret.
Differential Revision: https://phabricator.services.mozilla.com/D25001
--HG--
extra : moz-landing-system : lando
This only uses cross-platform tools, so switch to running these on linux, which
cuts the runtime down from ~20m to ~3m.
Differential Revision: https://phabricator.services.mozilla.com/D1080
--HG--
extra : moz-landing-system : lando
This is a hack to get around Windows ccov build hangs caused by bug 1195299.
Bug 1543149 will track the investigation of the hang and removal of this hack.
Differential Revision: https://phabricator.services.mozilla.com/D26750
--HG--
extra : moz-landing-system : lando
Run checks done in push-apk in promote-phase, instead of the very last task of the pipeline
Differential Revision: https://phabricator.services.mozilla.com/D26328
--HG--
rename : taskcluster/docker/google-play-strings/Dockerfile => taskcluster/docker/mozapkpublisher/Dockerfile
extra : moz-landing-system : lando
Quick fix for python3 mozbase perma-fail on osx: Use python 3.6 explicitly, rather
than the system default 3.7, which appears to be broken currently (lacking ssl support).
Differential Revision: https://phabricator.services.mozilla.com/D26345
--HG--
extra : moz-landing-system : lando
Ship-it v1 is going away soon and we won't need to create new releases in Ship-it v1 in parallel with Ship-it v2. It's time to prep patches to remove this functionality.
Differential Revision: https://phabricator.services.mozilla.com/D26044
--HG--
extra : moz-landing-system : lando
This makes sure that the mozharness scripts have access to all the packages in
the build system virtualenv (namely mozbase).
Differential Revision: https://phabricator.services.mozilla.com/D22184
--HG--
extra : moz-landing-system : lando
This makes sure that the mozharness scripts have access to all the packages in
the build system virtualenv (namely mozbase).
Differential Revision: https://phabricator.services.mozilla.com/D22184
--HG--
extra : moz-landing-system : lando
There was special case logic to map the win64 platform to win32, for stub
entries. When win64-aarch64 was added no special case was added for that
plaform, so they stub entries pointed at the incorrect place. This changes the
configuration so that all stub entries point at the win32 paths, without
needing special case code.
Differential Revision: https://phabricator.services.mozilla.com/D25841
--HG--
extra : moz-landing-system : lando
These will run on android-hw against the latest reference browser nightly.
Since they are try-only, they can only be scheduled with |mach try fuzzy
--full|.
Differential Revision: https://phabricator.services.mozilla.com/D25801
--HG--
extra : moz-landing-system : lando
We don't actually use the build-tools repo in-tree anymore, so remove the
support code for it.
Differential Revision: https://phabricator.services.mozilla.com/D25631
--HG--
extra : moz-landing-system : lando
This only uses cross-platform tools, so switch to running these on linux, which
cuts the runtime down from ~20m to ~3m.
Differential Revision: https://phabricator.services.mozilla.com/D1080
--HG--
extra : moz-landing-system : lando
Makes most kinds that reference nightly attribute reference the shippable attribute.
Also makes most transforms that use nightly use shippable
Transfers dependencies/ownership for some things to shippable from nightly where it was harder to support both.
In no particular order, full list:
Send shippable attribute down to dep tasks.
Set tests as shippable attribute
Add new signing routes
Add shippable routes to repackage_routes transform
Adjust target tasks
Add shippable nightly-l10n
Add nightly-signing and as a side affect add repackage and repackage-signing
Add support for proper balrog platforms for shippable
Add shippable to the nightly sccache guard
Fix TC_PLATFORM_PER_FTP in partners.py to use shippable
Add shippable to mozharness_test variants
Only actually used for android which doesn't have shippable at this time.
Add shippable variant to beetmover transforms
Do nightly signing for mars on shippable
Support shippable in partner-repack transform
Support shippable in amo langpacks transform
Use proper signing for shippable tasks in repackage transforms
Set upload symbols to use shippable too
Use shippable as deps for geckodriver extraction
Use shippable as dep for autograph-stage signing
Do not run beetmover-l10n for shippable
Run shippable style jobs for repackage signing
Set build_platform for update verify and uvc to be shippable
Run repackage-msi for shippable
Add shippable to osx partner repack signing
add shippable to emefree repackage
add shippable to emefree repackage signing
add shippable to beetmover checksums
Add shippable to partner repack repackage signing
add partner repack beetmover
Add shippable to mar signing
Add shippable to mar-signing-l10n
add shippable to eme free beetmover checksums
Add shippable to upload-generated-sources
Add beetmover langpacks to shippable
Add repackage-l10n to shippable
Add shippable to partner repack chunk-dummy
Do eme free builds with shippable
Add signing of language packs to shippable
Add snap repackage for shippable
Add shippable for release-eme-free repack signing
Add partials for shippable
Add partner repack repackage for shippable
Add emefree beetmover for shippable
Add checksums-signing to shippable
Switch partner repacks to shippable
Add shippable to beetmover-repackage
Add secondary update verify configs for shippable
secondary update verify for shippable
Differential Revision: https://phabricator.services.mozilla.com/D24699
--HG--
extra : moz-landing-system : lando
Makes most kinds that reference nightly attribute reference the shippable attribute.
Also makes most transforms that use nightly use shippable
Transfers dependencies/ownership for some things to shippable from nightly where it was harder to support both.
In no particular order, full list:
Send shippable attribute down to dep tasks.
Set tests as shippable attribute
Add new signing routes
Add shippable routes to repackage_routes transform
Adjust target tasks
Add shippable nightly-l10n
Add nightly-signing and as a side affect add repackage and repackage-signing
Add support for proper balrog platforms for shippable
Add shippable to the nightly sccache guard
Fix TC_PLATFORM_PER_FTP in partners.py to use shippable
Add shippable to mozharness_test variants
Only actually used for android which doesn't have shippable at this time.
Add shippable variant to beetmover transforms
Do nightly signing for mars on shippable
Support shippable in partner-repack transform
Support shippable in amo langpacks transform
Use proper signing for shippable tasks in repackage transforms
Set upload symbols to use shippable too
Use shippable as deps for geckodriver extraction
Use shippable as dep for autograph-stage signing
Do not run beetmover-l10n for shippable
Run shippable style jobs for repackage signing
Set build_platform for update verify and uvc to be shippable
Run repackage-msi for shippable
Add shippable to osx partner repack signing
add shippable to emefree repackage
add shippable to emefree repackage signing
add shippable to beetmover checksums
Add shippable to partner repack repackage signing
add partner repack beetmover
Add shippable to mar signing
Add shippable to mar-signing-l10n
add shippable to eme free beetmover checksums
Add shippable to upload-generated-sources
Add beetmover langpacks to shippable
Add repackage-l10n to shippable
Add shippable to partner repack chunk-dummy
Do eme free builds with shippable
Add signing of language packs to shippable
Add snap repackage for shippable
Add shippable for release-eme-free repack signing
Add partials for shippable
Add partner repack repackage for shippable
Add emefree beetmover for shippable
Add checksums-signing to shippable
Switch partner repacks to shippable
Add shippable to beetmover-repackage
Add secondary update verify configs for shippable
secondary update verify for shippable
Differential Revision: https://phabricator.services.mozilla.com/D24699
--HG--
extra : moz-landing-system : lando
Taskcluster has authorative information about the repository being built, so
pass that to mozharness, rather than have mozharness reconstruct it from
hand-maintained mozharness config.
Differential Revision: https://phabricator.services.mozilla.com/D24561
--HG--
extra : moz-landing-system : lando
The code that actually downloads it is behind a condition that isn't set
anywhere.
Differential Revision: https://phabricator.services.mozilla.com/D24215
--HG--
extra : moz-landing-system : lando
In bug 1501776, the `single_dep` and `multi_dep` schemas were aligned, which made both
branch in name_sanity identical. Simplify the code to reflect that.
Differential Revision: https://phabricator.services.mozilla.com/D24109
--HG--
extra : moz-landing-system : lando
test-linux.sh defaults to true for NEED_XVFB, while build-linux.sh
defaults to false. If we are using test-linux.sh from mozharness (rather
than mozharness-test), we need to explicitly set NEED_XVFB to false in
order to not use xvfb.
Differential Revision: https://phabricator.services.mozilla.com/D22820
--HG--
extra : source : 53d3443e55d95af494d6c8bdc3d2d7a52c5eff1e
test-linux.sh defaults to true for NEED_XVFB, while build-linux.sh
defaults to false. If we are using test-linux.sh from mozharness (rather
than mozharness-test), we need to explicitly set NEED_XVFB to false in
order to not use xvfb.
Differential Revision: https://phabricator.services.mozilla.com/D22820
--HG--
extra : moz-landing-system : lando
There are a number of ways we want to vary workers over time and jobs, including
- we are working on migrating to gce
- pgo builds have a dedicated worker-type for running the instrumented build
at level 3 but not level 1
Rather than have all tasks know about how the machines are provisioned, this
moves to using short-names for the worker types, and then has a config mapping
those to the actual worker types.
This adds support for aliases, and an initial set of them. Follow up work will
switch the existing uses of these worker types to using the aliases.
Differential Revision: https://phabricator.services.mozilla.com/D22549
--HG--
extra : moz-landing-system : lando
Bug 1486071 intended to fix this, but while the tasks are setup to
restart on exit status 100, there are multiple failure cases where
snapshot.debian.org doesn't respond and the exit status is not 100.
One is dget, when downloading package sources from snapshot.debian.org.
Eventually, those should move to fetch tasks, but in the meantime, we
bubble up get errors with an exit code 100.
mk-build-deps wraps a call to apt-get install, but doesn't return the
exit code that apt-get returns when apt-get returns one. It makes it
hard to distinguish the error modes, but mk-build-deps is unlikely to
fail on anything else than apt-get. Not all apt-get failures would be
due to snapshot.debian.org availability, but that's a tradeoff we
decided was okay in bug 1486071.
Differential Revision: https://phabricator.services.mozilla.com/D22269
--HG--
extra : moz-landing-system : lando
Currently the scopes are handled in some test-specific code. However, there is
logic not to be in generic code.
Differential Revision: https://phabricator.services.mozilla.com/D22447
--HG--
extra : moz-landing-system : lando
This slightly decreases the amount of code that needs to know how to determine this.
Differential Revision: https://phabricator.services.mozilla.com/D22446
--HG--
extra : moz-landing-system : lando
Currently the scopes are handled in some test-specific code. However, there is
logic not to be in generic code.
Differential Revision: https://phabricator.services.mozilla.com/D22447
--HG--
extra : moz-landing-system : lando
This slightly decreases the amount of code that needs to know how to determine this.
Differential Revision: https://phabricator.services.mozilla.com/D22446
--HG--
extra : moz-landing-system : lando
Currently, all tasks of kind builds are indiscriminately altered to use
artifacts, but only few of them actually support that, and the others
won't actually have the expected result when that happens. E.g. ASAN
builds with artifacts enabled end up being non-ASAN builds.
Effectively, this makes the artifact flag ignored for builds that don't
support artifacts. One could argue that those builds shouldn't happen at
all, but it feels a better use time of developer's time to just do the
full build they asked for. E.g. if they asked for ASAN with artifacts,
they still get an ASAN build, rather than an error or silently having
the task not happen after the decision task. This also allows to mix
artifact and non-artifact builds.
Further changes down the road are also modifying the artifact builds
configuration, which would actively turn those builds that don't support
artifact builds red (e.g. ASAN), so something has to be done anyways.
The alternative would be filter those builds out.
Depends on D21312
Differential Revision: https://phabricator.services.mozilla.com/D22056
--HG--
extra : moz-landing-system : lando
While try syntax is approaching its EOL, the fact that using it to do
artifact builds does some things subtly differently from using try
config is not helpful.
Depends on D22055
Differential Revision: https://phabricator.services.mozilla.com/D21312
--HG--
extra : moz-landing-system : lando
While the morph was changing the treeherder symbol to `Ba` for all jobs,
doing so with a transform fails because of the conflicting symbol check
(as multiple jobs in the same category would end up with `Ba`). So
instead, we append `a` to the existing symbol.
We also change the documentation wrt templates for try pushes, as the
artifact template is now essentially gone (although technically, mach
try will still set params['templates']['artifacts']['enabled'] for now,
and the template still exists, albeit empty).
Differential Revision: https://phabricator.services.mozilla.com/D22054
--HG--
extra : moz-landing-system : lando
We use autograph-prod for our ci, nightly, and release signing. Autograph-stage doesn't have the same guarantees for availability, so pointing, say, dep-signing at autograph-stage would have resulted in occasional tree closures whenever autograph-stage changes configuration or is down.
However, we also want a way to verify autograph-stage is still valid, after the autograph team makes changes. This task is meant to be add-task'ed; a green result means autograph-stage has signed the mar file correctly.
Differential Revision: https://phabricator.services.mozilla.com/D20749
--HG--
extra : moz-landing-system : lando
They appear to be causing tasks to take several hours to complete.
Differential Revision: https://phabricator.services.mozilla.com/D21775
--HG--
extra : rebase_source : a55cfc24527662427bbeccf0d03f97dca047a3cb
extra : amend_source : 5e352a1ff382353460fdd143d7d0ba52251a5b8a
A recent change caused the treeherder platform for several jobs to have an extra `/` in it.
This add a check to ensure that the platform is formatted correctly, and fixes the tasks
with the incorrect format.
Differential Revision: https://phabricator.services.mozilla.com/D21650
--HG--
extra : moz-landing-system : lando
Changes:
- increase maximum length for task identifiers to 38 from 22
- update documentation to state the same
Differential Revision: https://phabricator.services.mozilla.com/D20785
--HG--
extra : moz-landing-system : lando
We however leave moving the packages building to a script for another
day.
Differential Revision: https://phabricator.services.mozilla.com/D19624
--HG--
rename : taskcluster/docker/debian-base/cloud-mirror-workaround.sh => taskcluster/docker/debian-raw/cloud-mirror-workaround.sh
rename : taskcluster/docker/debian-base/setup_packages.sh => taskcluster/docker/debian-raw/setup_packages.sh
This code already exists for other (non-run-task jobs) on docker worker,
so this patch just reuses the existing code but adds a bit to the
run-task schema to allow the task to opt in.
Differential Revision: https://phabricator.services.mozilla.com/D19365
--HG--
extra : source : e9f887f489e59e828c7a62a4818c32cb5121f182
This code already exists for other (non-run-task jobs) on docker worker,
so this patch just reuses the existing code but adds a bit to the
run-task schema to allow the task to opt in.
Differential Revision: https://phabricator.services.mozilla.com/D19365
--HG--
extra : moz-landing-system : lando
- makes changes to various configuration files, thanks jmaher
- for the time being, disable tests via `taskcluster/ci/test/test-platforms.yml` to prevent overwhelming the hardware at Bitbar
Differential Revision: https://phabricator.services.mozilla.com/D19581
--HG--
extra : moz-landing-system : lando
This implements support for adding generic-worker caches. As a consequence this
also turns on caching for the gecko checkout if present.
Differential Revision: https://phabricator.services.mozilla.com/D17690
--HG--
extra : moz-landing-system : lando
We add caches at various places in common.py. This consolidates the logic into
a re-useable function. This is in preparation for adding generic-worker cache
support.
This also adds a test. The test is not terribly useful, but I've been looking
for an excuse to lay some groundwork for further tests in the 'job' submodule.
This will do.
Differential Revision: https://phabricator.services.mozilla.com/D17689
--HG--
extra : moz-landing-system : lando
This is useful for the out-of-tree taskgraph code. Downstream products can
pin the generated decision task image by revision, rather than contents.
Differential Revision: https://phabricator.services.mozilla.com/D19032
--HG--
extra : moz-landing-system : lando
This allows images to be built on every commit. This is useful for the
out-of-tree taskgraph, that builds a docker image with the taskgraph code
installed.
Differential Revision: https://phabricator.services.mozilla.com/D19031
--HG--
extra : moz-landing-system : lando
This implements support for adding generic-worker caches. As a consequence this
also turns on caching for the gecko checkout if present.
Differential Revision: https://phabricator.services.mozilla.com/D17690
--HG--
extra : moz-landing-system : lando
The hosts don't have enough disk space to cache mozilla-central.
Depends on D17689
Differential Revision: https://phabricator.services.mozilla.com/D18738
--HG--
extra : moz-landing-system : lando
We add caches at various places in common.py. This consolidates the logic into
a re-useable function. This is in preparation for adding generic-worker cache
support.
This also adds a test. The test is not terribly useful, but I've been looking
for an excuse to lay some groundwork for further tests in the 'job' submodule.
This will do.
Differential Revision: https://phabricator.services.mozilla.com/D17689
--HG--
extra : moz-landing-system : lando
The `asan-reporter` builds have shipping-product set on mozilla-central, so
that they get pulled in by the as dependencies of `post-beetmover-dummy`.
Differential Revision: https://phabricator.services.mozilla.com/D16580
--HG--
extra : moz-landing-system : lando
These builds aren't ready for general availability, so we don't want to publish
them. But we want to start building them now.
Differential Revision: https://phabricator.services.mozilla.com/D17453
--HG--
extra : moz-landing-system : lando
Chris AtLee
Ionut Goldan
shindli
Joel Maher
Rob Thijssen
Aki Sasaki
Andrew Halberstadt
Tom Prince
Mike Shal
Mike Hommey
Johan Lorenzo
Noemi Erli
alexandru.ionescu
Mitchell Hentges
Andrew Erickson
Bogdan Tara
Connor Sheehan
Daniel Varga
Chris Manchester
Geoff Brown
Rail Aliiev
Mihai Tabara
arthur.iakab
Ciure Andrei
Oana Pop Rus
Justin Wood
Dorel Luca
Sebastian Hengst
Andreea Pavel
Cosmin Sabou
Dustin J. Mitchell
Narcis Beleuzu
Gregory Mierzwinski
Brindusan Cristian
Simon Fraser
Nick Alexander
Nick Thomas
Bastien Abadie
Steve Fink
Byron Campen [:bwc]
Gurzau Raul
Edwin Gao
Kartikaya Gupta
Razvan Maries
David Major