A single task is created to do all partner attributions. The partner_attribution transform processes the configuration into an environment variable for the tools/attribution/attribute.py script to use. This is quite verbose so a large number of configurations may cause problems.
Applies the same priority modification to attribution tasks as to partner repacks, to not impede the main part of the graph.
Differential Revision: https://phabricator.services.mozilla.com/D87729
Renames the release_enable_partners parameter to release_enable_partner_repack, and adds release_enable_partner_attribution for attribution. This it to provide support for disabling them independently in main releases, and in respins.
Adds docs for attribution, update docs for repacks.
Hardwire values for the enable params for the respin flavors, other wise read from the input (defaulting to on in promotion, off otherwise).
Fixes up the rebuild-kinds for partner repacks so that they reflect the current set, although the top level may be all that is needed.
Differential Revision: https://phabricator.services.mozilla.com/D87727
I've left the monitor disabled for now, so that we can have a smaller pushes for enabling and disabling it if needed. It should allow more fine grained control.
We may also want to include extracting the monitor tool from a github version instead, and also removing the assumption and it being forked from the parent, so that it's instead given a process ID to treat as the parent it should watch.
Differential Revision: https://phabricator.services.mozilla.com/D84374
If we are generating only a part of the graph, to given kind, don't fail if a
build is packaging tests and there is no corresponding test task, as the tests
may not have been generated.
Differential Revision: https://phabricator.services.mozilla.com/D82097
Add an action that will trigger a task that runs
`mach release push-scriptworker-canary`
to test a new scriptworker deployment.
Differential Revision: https://phabricator.services.mozilla.com/D82821
Right now setup_perftest_test_date adds --test-date yesterday to all perftest
runs. we want that only for the ones doing batches
Differential Revision: https://phabricator.services.mozilla.com/D81562
This deliberately excludes builds that are implemented using the artifact build machinery,
but are not primarly intended to short-circuit build time. In particular the Windows aarch64
builds are not marked this way.
Differential Revision: https://phabricator.services.mozilla.com/D79549
Toolchains that are used for local development need to be built on a level-3
branch to installable via `mach bootstrap`. Add an attribute to track the fact
that a toolchain is used that way, and:
- ensure that everything installed via `mach boostrap` has that attribute set
- ensure that everything with that attribute set is built on trunk projects
We could additionally verify that attribute is only set on things used by
bootstrap, but bootstrap doesn't currently have an exhaustive list of things
that it might install, making that difficult.
Differential Revision: https://phabricator.services.mozilla.com/D77706
Currently test manifests are loaded by instantiating a TestResolver and
traversing moz.build files to find manifests.
With 'manifest-scheduling', we'll want to grab the manifests directly from the
bugbug service instead. Initially we'll want to enable manifest-scheduling with
|mach try auto|, but not on autoland yet.
This patch will allow |mach try auto| to set the parameter that causes bugbug
to be used (see future commits in this bug).
Differential Revision: https://phabricator.services.mozilla.com/D76522
This patch is adding an option to push a perftest run in the CI.
It's based on :
- sparse profiles
- push_to_try
- options passed through try_task_config.json
Differential Revision: https://phabricator.services.mozilla.com/D74115
Doing this in the ship phase instead of push lets us avoid shipping on
flathub before the actual release.
And, upload flatpaks for firefox release candidates to flathub's "beta"
channel so we can get feedback on them and QA can also get at them.
Differential Revision: https://phabricator.services.mozilla.com/D71919
Add nazgul counterpart for bouncer-locations
Add nazgul counterpart for partnew-repack-bouncer-sub
Differential Revision: https://phabricator.services.mozilla.com/D67887
--HG--
extra : moz-landing-system : lando
The 'auto' in 'mach try auto' stands for two things:
1. It automatically picks tasks for you.
2. It runs the same scheduling algorithms as autoland.
It accomplishes this by creating a new target_tasks method that spoofs the
'project' parameter to autoland.
Differential Revision: https://phabricator.services.mozilla.com/D60184
--HG--
extra : moz-landing-system : lando
Adjust runme to be executable
Rm sdk/pltf install.Manually add meta
Enforce firefox run instead of notify
Add policy to disable updates
Temp hack to default to firefox instead of notify-send
Fix mach linters
Remove firefox command hack. Proper fix
Remove duplicate cmd in runme
Fix indentantion in kind
Fix more linters
Differential Revision: https://phabricator.services.mozilla.com/D59561
--HG--
extra : moz-landing-system : lando
Calling the merge day automation requires an action so that we can pass in parameters such as source and destination repository and branch.
Differential Revision: https://phabricator.services.mozilla.com/D62763
--HG--
extra : moz-landing-system : lando
We don't need to package tests for builds that we don't actually run
tests from, but it is tricky to align this correctly by setting
MOZ_AUTOMATION_PACKAGE_TESTS=0 in relevant mozconfigs. Instead we can
set the environment variable in the task definition, and use a full
taskgraph verification check to ensure that the flag is only set on
builds that have tests.
The one tricky area is the win64-aarch64 builds, which have a workaround
by specifying the new skip-verify-test-packaging attribute.
In one case, win64-aarch64-shippable has tests that run against it, but
it copies those tests from a win64-aarch64-shippable-no-eme task, which
itself has no tests. Both of those tasks need to skip the verify check
as a result.
In another case, the win64-aarch64-eme task is an artifact build that
grabs test packages from the win64-aarch64 build. Since the win64-aarch64
build doesn't have tests, it needs to skip the verify check.
Differential Revision: https://phabricator.services.mozilla.com/D59426
--HG--
extra : moz-landing-system : lando
We don't need to package tests for builds that we don't actually run
tests from, but it is tricky to align this correctly by setting
MOZ_AUTOMATION_PACKAGE_TESTS=0 in relevant mozconfigs. Instead we can
set the environment variable in the task definition, and use a full
taskgraph verification check to ensure that the flag is only set on
builds that have tests.
The one tricky area is the win64-aarch64 builds, which have a workaround
by specifying the new skip-verify-test-packaging attribute.
In one case, win64-aarch64-shippable has tests that run against it, but
it copies those tests from a win64-aarch64-shippable-no-eme task, which
itself has no tests. Both of those tasks need to skip the verify check
as a result.
In another case, the win64-aarch64-eme task is an artifact build that
grabs test packages from the win64-aarch64 build. Since the win64-aarch64
build doesn't have tests, it needs to skip the verify check.
Differential Revision: https://phabricator.services.mozilla.com/D59426
--HG--
extra : moz-landing-system : lando
It will be relanded once these are complete. This prevents from those tasks
getting scheduled for every push until the initial ones have been completed.
This change introduces a "github-sync" component into tools,
which aims to support synchronizing both wgpu and WebRender with github.
~~It also features a "cargo test" job for standalone wgpu (bug 1596127)~~
The code is ported from "gfx/wr/scripts/wrupdater" folder. Changes are:
1. remove explicit WR parts and make them configurable by command line params
2. detect "mozilla-xxx" tags and use them in addition to the commits
As a follow up, wrupdater will be removed in favor of github-sync.
Status:
- [x] get the CI test job working
- [x] get @kats to fork "wgpu" github for "moz-gfx" bot
- [x] remove the wgpu testing CI job (into separate PR)
- [x] create new secret and reference it
Differential Revision: https://phabricator.services.mozilla.com/D57057
--HG--
extra : moz-landing-system : lando
We've long handled chunks by defining the total number of chunks in our CI
configuration, and then passing that value down into the test harnesses at task
runtime (via the '--this-chunk' and '--total-chunks' parameters). The test
harness then runs an algorithm to determine which tests should be run in "this"
chunk.
There are several problems with this approach, but by far the biggest is that
we can't use test information in our scheduling algorithms. The information
simply isn't available yet. This patch switches things around such that we
determine which tests go in which tasks during the taskgraph generation. This
means we have perfect information around which tasks are running which tests,
and if e.g a ccov or machine learning algorithm deems a particular test
important, we can make sure to *only* schedule the tasks that contain that
test.
I'm planning to enable this a couple suites at a time so we don't accidentally
stop running tests. This specifically only enables this mode for
'mochitest-media', 'mochitest-browser-chrome' and 'mochitest-devtools-chrome'.
I chose these suites because they are the ones that are already using the
'chunk_by_runtime' algorithm.
Differential Revision: https://phabricator.services.mozilla.com/D52729
--HG--
extra : moz-landing-system : lando
Handles 'env' and 'chemspill-prio' configs in the transforms. The 'rebuild'
task config is purposefully excluded from the full_task_graph and instead
handled at the target_tasks stage. Otherwise if a user ran '--rebuild 20' then
retriggered one of those tasks, they'd instead get another 20 which is almost
certainly not what we want.
Differential Revision: https://phabricator.services.mozilla.com/D51417
--HG--
extra : moz-landing-system : lando
This patch enables the iris test suite to run in CI against Windows and Linux shippable builds on mozilla-central and try. The framework is in place for Iris to run against MacOS in CI, but it is currently disabled while bootstrapping issues are sorted out.
Linux uses a new docker image based on the debian10-test parent image that installs preinstalls most of Iris's dependencies. Windows installs a few dependencies using the scoop package manager. Both then install the rest of the python dependencies via pip.
This adds a new toolchain artifact to fetch the iris_firefox git repo without touching the outside network.
Differential Revision: https://phabricator.services.mozilla.com/D41638
--HG--
extra : moz-landing-system : lando
This new task fetches the visualmetrics.py script from the
github.com/mozilla/browsertime repository and runs it in parallel for the
specified jobs. Jobs are specified in a JSON blob passed through to the task in
an environment variable. A follow up patch specifies a command line argument to
make this configuration available to `./mach try {fuzzy|chooser}`
Differential Revision: https://phabricator.services.mozilla.com/D41052
--HG--
extra : moz-landing-system : lando
This new task fetches the visualmetrics.py script from the
github.com/mozilla/browsertime repository and runs it in parallel for the
specified jobs. Jobs are specified in a JSON blob passed through to the task in
an environment variable. A follow up patch specifies a command line argument to
make this configuration available to `./mach try {fuzzy|chooser}`
Differential Revision: https://phabricator.services.mozilla.com/D41052
--HG--
extra : moz-landing-system : lando
Duplicating the old bouncer submission task and having it point to Nazgul in both stage and prod.
Differential Revision: https://phabricator.services.mozilla.com/D42171
--HG--
extra : moz-landing-system : lando
Sign language packs via Autograph instead of AMO.
Differential Revision: https://phabricator.services.mozilla.com/D38148
--HG--
rename : taskcluster/ci/release-sign-and-push-langpacks/kind.yml => taskcluster/ci/release-push-langpacks/kind.yml
extra : moz-landing-system : lando
Sign language packs via Autograph instead of AMO.
Differential Revision: https://phabricator.services.mozilla.com/D38148
--HG--
rename : taskcluster/ci/release-sign-and-push-langpacks/kind.yml => taskcluster/ci/release-push-langpacks/kind.yml
extra : moz-landing-system : lando
Bug 1563711 changed MOZ_FETCHES_DIR to make the MOZ_ANDROID_FAT_AAR_*
environment variables absolute paths. Unfortunately, the replacement
relies on non-osx/windows tasks to run in docker-worker, which is not
necessarily true, and that makes the MOZ_FETCHES_DIR wrong in
non-osx/windows generic-worker tasks. Apparently, that currently works,
but that's not guaranteed to stay this way.
The MOZ_ANDROID_FAT_AAR_* environment variables don't need to be
absolute paths, though. MOZ_FETCHES_DIR is normalized by run-task, and
MOZ_ANDROID_FAT_AAR_* can be set relative to that, which we do here.
Differential Revision: https://phabricator.services.mozilla.com/D40334
Our datadog contract is coming to an end. Soon we'll add influxdb data collection
Differential Revision: https://phabricator.services.mozilla.com/D38086
--HG--
extra : moz-landing-system : lando
Taskgraph needs to know the correct mar-channel, so allow it to pass it into the build,
rather than keying off the update-channel in configure. This will allow using a `mar`
binary that doesn't have the mar-channel configured in.
Differential Revision: https://phabricator.services.mozilla.com/D37480
--HG--
extra : moz-landing-system : lando
Adds a new release-partner-repack-bouncer-sub-firefox task to add the partner products. This is forked from the main bouncer submission because it diverges sufficiently that this is a cleaner approach. The get_partners_to_be_published helper is used by aliases too.
Depends on D34950
Differential Revision: https://phabricator.services.mozilla.com/D35481
--HG--
extra : moz-landing-system : lando
This adds a new set of update tasks and channels for testing updates from the
previous ESR release, before we make the new release generally available as an
update.
Differential Revision: https://phabricator.services.mozilla.com/D31654
--HG--
extra : moz-landing-system : lando
Currently we have the concept of a "suite" and a "flavour" in our task
configuration. Typically, the "suite" refers to the high-level test harness
like "mochitest" or "reftest", whereas the flavour is more specific, e.g
"browser-chrome-instrumentation" or "crashtest". However the line between suite
and flavour is not applied with any semblance of consistency which results in
inconsistent naming throughout the tree.
This patch gets rid of the concept of "flavours" entirely (at least when it
comes to task configuration). A suite is a type of test run, for example:
- mochitest-plain
- mochitest-devtools-chrome
- mochitest-browser-chrome-instrumentation
- jsreftest
- reftest
- firefox-ui-functional-remote
etc
There is no confusion here between suites and flavours because flavours don't
exist. However, there are a couple of places where we *do* need to know what
"test harness" is used to run a suite. These cases are:
1. For SCHEDULES moz.build rules
2. For the desktop_unittest.py mozharness script which takes arguments like
--mochitest-suite=browser (this is not a compelling use of this information
and should be refactored to work more like the android_emulator_unittest.py
script)
So to get this information, this patch introduces a new concept of a "category"
which is the overall "test harness" that runs the suite. For many suites, the
"category" is identical to the suite name. Unlike flavours, "categories" have
no bearing on how we call or refer to the suite.
Differential Revision: https://phabricator.services.mozilla.com/D27554
--HG--
extra : moz-landing-system : lando
We are starting to spin off more and more "variants" of test suites. These are
usually just duplicates of our pre-existing tasks, except with an additional
pref set.
Currently there are two variants (serviceworker-e10s and socketprocess-e10s),
but a third will be added soon (fission). This change ensures we handle these
types of requests in a consistent and well defined manner. It also splits tasks
in a loop, so we don't accidentally risk combinatorial explosion.
Variants should typically be reserved for very large changes that will impact
the entire codebase (think e10s).
Differential Revision: https://phabricator.services.mozilla.com/D28061
--HG--
extra : moz-landing-system : lando
Many tasks (release tasks and cached tasks, in particular) should be re-run rather
than retriggered. Disable retrigger action for those tasks by default.
Differential Revision: https://phabricator.services.mozilla.com/D27206
--HG--
extra : moz-landing-system : lando
Run checks done in push-apk in promote-phase, instead of the very last task of the pipeline
Differential Revision: https://phabricator.services.mozilla.com/D26328
--HG--
rename : taskcluster/docker/google-play-strings/Dockerfile => taskcluster/docker/mozapkpublisher/Dockerfile
extra : moz-landing-system : lando
Ship-it v1 is going away soon and we won't need to create new releases in Ship-it v1 in parallel with Ship-it v2. It's time to prep patches to remove this functionality.
Differential Revision: https://phabricator.services.mozilla.com/D26044
--HG--
extra : moz-landing-system : lando
When we set the nightly attribute the tasks don't run on-push, so we use a new attribute.
Differential Revision: https://phabricator.services.mozilla.com/D22709
--HG--
extra : moz-landing-system : lando
Small updates to the version control documents regarding
upgrading `robustcheckout.py`. I noticed that we still
reference `build-puppet` from it's former home on hgmo
and don't include OpenCloudConfig in our list of places
we need to vendor `robustcheckout` changes.
Differential Revision: https://phabricator.services.mozilla.com/D23742
--HG--
extra : moz-landing-system : lando
This task extracts the binary of geckodriver from the
common test package and stores it into its own artifact.
For now this task is only run after Nightly build tasks
on supported platforms..
Differential Revision: https://phabricator.services.mozilla.com/D23094
--HG--
extra : moz-landing-system : lando
While the morph was changing the treeherder symbol to `Ba` for all jobs,
doing so with a transform fails because of the conflicting symbol check
(as multiple jobs in the same category would end up with `Ba`). So
instead, we append `a` to the existing symbol.
We also change the documentation wrt templates for try pushes, as the
artifact template is now essentially gone (although technically, mach
try will still set params['templates']['artifacts']['enabled'] for now,
and the template still exists, albeit empty).
Differential Revision: https://phabricator.services.mozilla.com/D22054
--HG--
extra : moz-landing-system : lando