Calling the merge day automation requires an action so that we can pass in parameters such as source and destination repository and branch.
Differential Revision: https://phabricator.services.mozilla.com/D62763
--HG--
extra : moz-landing-system : lando
We don't need to package tests for builds that we don't actually run
tests from, but it is tricky to align this correctly by setting
MOZ_AUTOMATION_PACKAGE_TESTS=0 in relevant mozconfigs. Instead we can
set the environment variable in the task definition, and use a full
taskgraph verification check to ensure that the flag is only set on
builds that have tests.
The one tricky area is the win64-aarch64 builds, which have a workaround
by specifying the new skip-verify-test-packaging attribute.
In one case, win64-aarch64-shippable has tests that run against it, but
it copies those tests from a win64-aarch64-shippable-no-eme task, which
itself has no tests. Both of those tasks need to skip the verify check
as a result.
In another case, the win64-aarch64-eme task is an artifact build that
grabs test packages from the win64-aarch64 build. Since the win64-aarch64
build doesn't have tests, it needs to skip the verify check.
Differential Revision: https://phabricator.services.mozilla.com/D59426
--HG--
extra : moz-landing-system : lando
We don't need to package tests for builds that we don't actually run
tests from, but it is tricky to align this correctly by setting
MOZ_AUTOMATION_PACKAGE_TESTS=0 in relevant mozconfigs. Instead we can
set the environment variable in the task definition, and use a full
taskgraph verification check to ensure that the flag is only set on
builds that have tests.
The one tricky area is the win64-aarch64 builds, which have a workaround
by specifying the new skip-verify-test-packaging attribute.
In one case, win64-aarch64-shippable has tests that run against it, but
it copies those tests from a win64-aarch64-shippable-no-eme task, which
itself has no tests. Both of those tasks need to skip the verify check
as a result.
In another case, the win64-aarch64-eme task is an artifact build that
grabs test packages from the win64-aarch64 build. Since the win64-aarch64
build doesn't have tests, it needs to skip the verify check.
Differential Revision: https://phabricator.services.mozilla.com/D59426
--HG--
extra : moz-landing-system : lando
It will be relanded once these are complete. This prevents from those tasks
getting scheduled for every push until the initial ones have been completed.
This change introduces a "github-sync" component into tools,
which aims to support synchronizing both wgpu and WebRender with github.
~~It also features a "cargo test" job for standalone wgpu (bug 1596127)~~
The code is ported from "gfx/wr/scripts/wrupdater" folder. Changes are:
1. remove explicit WR parts and make them configurable by command line params
2. detect "mozilla-xxx" tags and use them in addition to the commits
As a follow up, wrupdater will be removed in favor of github-sync.
Status:
- [x] get the CI test job working
- [x] get @kats to fork "wgpu" github for "moz-gfx" bot
- [x] remove the wgpu testing CI job (into separate PR)
- [x] create new secret and reference it
Differential Revision: https://phabricator.services.mozilla.com/D57057
--HG--
extra : moz-landing-system : lando
We've long handled chunks by defining the total number of chunks in our CI
configuration, and then passing that value down into the test harnesses at task
runtime (via the '--this-chunk' and '--total-chunks' parameters). The test
harness then runs an algorithm to determine which tests should be run in "this"
chunk.
There are several problems with this approach, but by far the biggest is that
we can't use test information in our scheduling algorithms. The information
simply isn't available yet. This patch switches things around such that we
determine which tests go in which tasks during the taskgraph generation. This
means we have perfect information around which tasks are running which tests,
and if e.g a ccov or machine learning algorithm deems a particular test
important, we can make sure to *only* schedule the tasks that contain that
test.
I'm planning to enable this a couple suites at a time so we don't accidentally
stop running tests. This specifically only enables this mode for
'mochitest-media', 'mochitest-browser-chrome' and 'mochitest-devtools-chrome'.
I chose these suites because they are the ones that are already using the
'chunk_by_runtime' algorithm.
Differential Revision: https://phabricator.services.mozilla.com/D52729
--HG--
extra : moz-landing-system : lando
Handles 'env' and 'chemspill-prio' configs in the transforms. The 'rebuild'
task config is purposefully excluded from the full_task_graph and instead
handled at the target_tasks stage. Otherwise if a user ran '--rebuild 20' then
retriggered one of those tasks, they'd instead get another 20 which is almost
certainly not what we want.
Differential Revision: https://phabricator.services.mozilla.com/D51417
--HG--
extra : moz-landing-system : lando
This patch enables the iris test suite to run in CI against Windows and Linux shippable builds on mozilla-central and try. The framework is in place for Iris to run against MacOS in CI, but it is currently disabled while bootstrapping issues are sorted out.
Linux uses a new docker image based on the debian10-test parent image that installs preinstalls most of Iris's dependencies. Windows installs a few dependencies using the scoop package manager. Both then install the rest of the python dependencies via pip.
This adds a new toolchain artifact to fetch the iris_firefox git repo without touching the outside network.
Differential Revision: https://phabricator.services.mozilla.com/D41638
--HG--
extra : moz-landing-system : lando
This new task fetches the visualmetrics.py script from the
github.com/mozilla/browsertime repository and runs it in parallel for the
specified jobs. Jobs are specified in a JSON blob passed through to the task in
an environment variable. A follow up patch specifies a command line argument to
make this configuration available to `./mach try {fuzzy|chooser}`
Differential Revision: https://phabricator.services.mozilla.com/D41052
--HG--
extra : moz-landing-system : lando
This new task fetches the visualmetrics.py script from the
github.com/mozilla/browsertime repository and runs it in parallel for the
specified jobs. Jobs are specified in a JSON blob passed through to the task in
an environment variable. A follow up patch specifies a command line argument to
make this configuration available to `./mach try {fuzzy|chooser}`
Differential Revision: https://phabricator.services.mozilla.com/D41052
--HG--
extra : moz-landing-system : lando
Duplicating the old bouncer submission task and having it point to Nazgul in both stage and prod.
Differential Revision: https://phabricator.services.mozilla.com/D42171
--HG--
extra : moz-landing-system : lando
Sign language packs via Autograph instead of AMO.
Differential Revision: https://phabricator.services.mozilla.com/D38148
--HG--
rename : taskcluster/ci/release-sign-and-push-langpacks/kind.yml => taskcluster/ci/release-push-langpacks/kind.yml
extra : moz-landing-system : lando
Sign language packs via Autograph instead of AMO.
Differential Revision: https://phabricator.services.mozilla.com/D38148
--HG--
rename : taskcluster/ci/release-sign-and-push-langpacks/kind.yml => taskcluster/ci/release-push-langpacks/kind.yml
extra : moz-landing-system : lando
Bug 1563711 changed MOZ_FETCHES_DIR to make the MOZ_ANDROID_FAT_AAR_*
environment variables absolute paths. Unfortunately, the replacement
relies on non-osx/windows tasks to run in docker-worker, which is not
necessarily true, and that makes the MOZ_FETCHES_DIR wrong in
non-osx/windows generic-worker tasks. Apparently, that currently works,
but that's not guaranteed to stay this way.
The MOZ_ANDROID_FAT_AAR_* environment variables don't need to be
absolute paths, though. MOZ_FETCHES_DIR is normalized by run-task, and
MOZ_ANDROID_FAT_AAR_* can be set relative to that, which we do here.
Differential Revision: https://phabricator.services.mozilla.com/D40334
Our datadog contract is coming to an end. Soon we'll add influxdb data collection
Differential Revision: https://phabricator.services.mozilla.com/D38086
--HG--
extra : moz-landing-system : lando
Taskgraph needs to know the correct mar-channel, so allow it to pass it into the build,
rather than keying off the update-channel in configure. This will allow using a `mar`
binary that doesn't have the mar-channel configured in.
Differential Revision: https://phabricator.services.mozilla.com/D37480
--HG--
extra : moz-landing-system : lando
Adds a new release-partner-repack-bouncer-sub-firefox task to add the partner products. This is forked from the main bouncer submission because it diverges sufficiently that this is a cleaner approach. The get_partners_to_be_published helper is used by aliases too.
Depends on D34950
Differential Revision: https://phabricator.services.mozilla.com/D35481
--HG--
extra : moz-landing-system : lando
This adds a new set of update tasks and channels for testing updates from the
previous ESR release, before we make the new release generally available as an
update.
Differential Revision: https://phabricator.services.mozilla.com/D31654
--HG--
extra : moz-landing-system : lando
Currently we have the concept of a "suite" and a "flavour" in our task
configuration. Typically, the "suite" refers to the high-level test harness
like "mochitest" or "reftest", whereas the flavour is more specific, e.g
"browser-chrome-instrumentation" or "crashtest". However the line between suite
and flavour is not applied with any semblance of consistency which results in
inconsistent naming throughout the tree.
This patch gets rid of the concept of "flavours" entirely (at least when it
comes to task configuration). A suite is a type of test run, for example:
- mochitest-plain
- mochitest-devtools-chrome
- mochitest-browser-chrome-instrumentation
- jsreftest
- reftest
- firefox-ui-functional-remote
etc
There is no confusion here between suites and flavours because flavours don't
exist. However, there are a couple of places where we *do* need to know what
"test harness" is used to run a suite. These cases are:
1. For SCHEDULES moz.build rules
2. For the desktop_unittest.py mozharness script which takes arguments like
--mochitest-suite=browser (this is not a compelling use of this information
and should be refactored to work more like the android_emulator_unittest.py
script)
So to get this information, this patch introduces a new concept of a "category"
which is the overall "test harness" that runs the suite. For many suites, the
"category" is identical to the suite name. Unlike flavours, "categories" have
no bearing on how we call or refer to the suite.
Differential Revision: https://phabricator.services.mozilla.com/D27554
--HG--
extra : moz-landing-system : lando
We are starting to spin off more and more "variants" of test suites. These are
usually just duplicates of our pre-existing tasks, except with an additional
pref set.
Currently there are two variants (serviceworker-e10s and socketprocess-e10s),
but a third will be added soon (fission). This change ensures we handle these
types of requests in a consistent and well defined manner. It also splits tasks
in a loop, so we don't accidentally risk combinatorial explosion.
Variants should typically be reserved for very large changes that will impact
the entire codebase (think e10s).
Differential Revision: https://phabricator.services.mozilla.com/D28061
--HG--
extra : moz-landing-system : lando
Many tasks (release tasks and cached tasks, in particular) should be re-run rather
than retriggered. Disable retrigger action for those tasks by default.
Differential Revision: https://phabricator.services.mozilla.com/D27206
--HG--
extra : moz-landing-system : lando
Run checks done in push-apk in promote-phase, instead of the very last task of the pipeline
Differential Revision: https://phabricator.services.mozilla.com/D26328
--HG--
rename : taskcluster/docker/google-play-strings/Dockerfile => taskcluster/docker/mozapkpublisher/Dockerfile
extra : moz-landing-system : lando
Ship-it v1 is going away soon and we won't need to create new releases in Ship-it v1 in parallel with Ship-it v2. It's time to prep patches to remove this functionality.
Differential Revision: https://phabricator.services.mozilla.com/D26044
--HG--
extra : moz-landing-system : lando
When we set the nightly attribute the tasks don't run on-push, so we use a new attribute.
Differential Revision: https://phabricator.services.mozilla.com/D22709
--HG--
extra : moz-landing-system : lando
Small updates to the version control documents regarding
upgrading `robustcheckout.py`. I noticed that we still
reference `build-puppet` from it's former home on hgmo
and don't include OpenCloudConfig in our list of places
we need to vendor `robustcheckout` changes.
Differential Revision: https://phabricator.services.mozilla.com/D23742
--HG--
extra : moz-landing-system : lando
This task extracts the binary of geckodriver from the
common test package and stores it into its own artifact.
For now this task is only run after Nightly build tasks
on supported platforms..
Differential Revision: https://phabricator.services.mozilla.com/D23094
--HG--
extra : moz-landing-system : lando
While the morph was changing the treeherder symbol to `Ba` for all jobs,
doing so with a transform fails because of the conflicting symbol check
(as multiple jobs in the same category would end up with `Ba`). So
instead, we append `a` to the existing symbol.
We also change the documentation wrt templates for try pushes, as the
artifact template is now essentially gone (although technically, mach
try will still set params['templates']['artifacts']['enabled'] for now,
and the template still exists, albeit empty).
Differential Revision: https://phabricator.services.mozilla.com/D22054
--HG--
extra : moz-landing-system : lando
Simon Fraser
Aki Sasaki
Mike Shal
Andreea Pavel
Dorel Luca
Justin Wood
Noemi Erli
Dzmitry Malyshau
Ryan VanderMeulen
Sebastian Hengst
Tom Prince
Andrew Halberstadt
Mihai Tabara
Sylvestre Ledru
Ciure Andrei
Tarek Ziadé
Nick Thomas
Wes Kocher
Chris AtLee
Barret Rennie
Narcis Beleuzu
Dustin J. Mitchell
Aaron Amanuel
championshuttler
Mike Hommey
Johan Lorenzo
Nick Alexander
shindli
Rail Aliiev
Connor Sheehan
Henrik Skupin
Brindusan Cristian
Bastien Abadie