We need this bump for a couple of reasons:
1. It enables floating the libc crate version, which makes keeping up-to-date
with upstream changes easier.
2. It enables floating the getrandom crate version, which enables updating
getrandom to a version with less buggy behavior around the Linux
`getrandom` syscall on some architectures.
We also take this opportunity to sync up our vendored lucet with the lucet
that we use to build things in taskcluster.
Differential Revision: https://phabricator.services.mozilla.com/D85408
Sets autoland to use the 'bugbug' test manifest loader. This is being enabled
as part of a temporary trial to see the impact it has on sheriffing.
Differential Revision: https://phabricator.services.mozilla.com/D85011
The enhancements in Bug 1653986 to display information about tests derives its
data from these 2 jobs and so it's appropriate to explicitly depend on them.
The current status of these jobs in the tree as far as I can tell is that:
- "source-test-file-metadata-test-info-all": Seems to get automatically
triggered by someone's regularly, so it's always likely to be there for
our searchfox cron jobs, but it's nice to not depend on that.
- "source-test-wpt-metadata-summary" : Seems to get automatically run, but
only on commits that touch meta files change, per
https://searchfox.org/mozilla-central/rev/cf561cece0ca9aeaf0202e68699836d957d0c670/taskcluster/ci/source-test/wpt-metadata.yml#31
and indeed it wasn't there on today's searchfox jobs.
Differential Revision: https://phabricator.services.mozilla.com/D84933
Sets autoland to use the 'bugbug' test manifest loader. This is being enabled
as part of a temporary trial to see the impact it has on sheriffing.
Differential Revision: https://phabricator.services.mozilla.com/D85011
This creates a new set of optimization strategies
(taskgraph.optimize:project.autoland) to use with autoland. Among other things,
it also means there's no need for the 'test-try' optimization as the autoland
strategies are no longer the default behaviour.
Differential Revision: https://phabricator.services.mozilla.com/D79705
Creates a new utility function in 'taskgraph.util.taskcluster' to return task
definitions from a task group. Also refactors the existing
'list_task_group_incomplete_tasks' function to use it.
Differential Revision: https://phabricator.services.mozilla.com/D74407
Windows jobs typically take 23 minutes, so limit to 30 minutes so that cloning hangs terminate earlier than 2 hours.
Mac and Linux jobs are typically 5 minutes long, so bring the limit in to 15 minutes from an hour.
Also refactors so this the max run time is only specified in the kind, rather than some in the kind and some in the transform.
Differential Revision: https://phabricator.services.mozilla.com/D85077
This adds a makecab toolchain for Linux. It's not hooked anywhere
because bug 1654994 will also move the use of makecab to upload-symbols
tasks, so hooking the toolchain up with builds would be a waste of
time.
Differential Revision: https://phabricator.services.mozilla.com/D84789
It could go into its own test suite, but it 1) depends on `mozbuild` code, so the `mozbuild` suite as well as this new suite would be running on any push that touches `mozbuild` code anyway, and 2) this is code that runs during the build, so it's not out of place.
Differential Revision: https://phabricator.services.mozilla.com/D84547
It could go into its own test suite, but it 1) depends on `mozbuild` code, so the `mozbuild` suite as well as this new suite would be running on any push that touches `mozbuild` code anyway, and 2) this is code that runs during the build, so it's not out of place.
Differential Revision: https://phabricator.services.mozilla.com/D84547
The Reference Browser represents our tech stack up to and including the Android-Components layer.
It would be useful to have an applink startup test for this browser to detect improvements and regressions.
Differential Revision: https://phabricator.services.mozilla.com/D81407
This creates a new set of optimization strategies
(taskgraph.optimize:project.autoland) to use with autoland. Among other things,
it also means there's no need for the 'test-try' optimization as the autoland
strategies are no longer the default behaviour.
Differential Revision: https://phabricator.services.mozilla.com/D79705
Creates a new utility function in 'taskgraph.util.taskcluster' to return task
definitions from a task group. Also refactors the existing
'list_task_group_incomplete_tasks' function to use it.
Differential Revision: https://phabricator.services.mozilla.com/D74407
It could go into its own test suite, but it 1) depends on `mozbuild` code, so the `mozbuild` suite as well as this new suite would be running on any push that touches `mozbuild` code anyway, and 2) this is code that runs during the build, so it's not out of place.
Differential Revision: https://phabricator.services.mozilla.com/D84547
I suspect these were an artifact of building the checked-out repository
inside the Firefox source directory, but that is not a problem anymore.
Differential Revision: https://phabricator.services.mozilla.com/D84363
We don't need this for our (current) builds, which are cross-builds, but we
would need this at some future date if we ditched the breakpad `dump_syms`.
Depends on D83528
Differential Revision: https://phabricator.services.mozilla.com/D83529
I suspect these were an artifact of building the checked-out repository
inside the Firefox source directory, but that is not a problem anymore.
Differential Revision: https://phabricator.services.mozilla.com/D83528
When links to specific tasks in treeherder was added to the description in
Bug 1630710, This caused the descriptions for several derived tasks to include
the `{'task-reference': '...'}` wrapper in the derived description. This is
in addition to the already duplicate links to treeherder those tasks had.
To ameliorate this issue, keep track of the original description without the
treeherder link and use that in derived tasks.
Differential Revision: https://phabricator.services.mozilla.com/D84047
This adds the talos-profiling jobs to the relevant test-platforms so that they
can be run for Linux and Windows shippable builds with WebRender enabled.
However they are also disabled from running on m-c by default; they must be
run explicitly using `./mach try fuzzy --full` or using the TreeHerder add jobs
feature.
Differential Revision: https://phabricator.services.mozilla.com/D84011
This block is identical to the run-on-projects applied to all the talos
profiling jobs in the defaults section at the top of the file. Therefore
it is redundant.
Differential Revision: https://phabricator.services.mozilla.com/D84010
Since the system mozdevice version on bitbar is old, and mozharness now requires
mozdevice 4.0.0, we simply have the build job make a tarball of mozdevice 4.0.0
from the gecko source tree and export it as an artifact. The wrench job that
runs on bitbar can unpack the tarball and add it to the python path so it gets
picked up.
Differential Revision: https://phabricator.services.mozilla.com/D84070
This change the strategy aliases that have bugbug, so that they handle the extra
arguments that are needed by bugbug.
Differential Revision: https://phabricator.services.mozilla.com/D83438
This patch adds a new pageload testing mode called chimera mode. This mode runs the cold and warm pageload tests together within a single browser session (cold first, warm second). Some changes had to be made in the visual metrics task to be able to do this (per-test specification of extraOptions).
Differential Revision: https://phabricator.services.mozilla.com/D83356
This change the strategy aliases that have bugbug, so that they handle the extra
arguments that are needed by bugbug.
Differential Revision: https://phabricator.services.mozilla.com/D83438
I'm planning on replacing voluptuous with something faster in Bug 1652123, but
in the meantime, compile the schema is slow, so there is no reason to do so if
we aren't going to check them anyway.
Differential Revision: https://phabricator.services.mozilla.com/D83540
Changes:
- create a `test-android.sh` script that is based on, but removes non-android related paths from the `test-linux.sh` script.
Differential Revision: https://phabricator.services.mozilla.com/D82252
Changes:
- set up a new minimal docker image for `android-em` tests.
- reorganize commonly referenced files into a specific folder `taskcluster/docker/files` to reduce instances of same file being copy-pasted into directories.
- add android-test docker build task to taskcluster.
Differential Revision: https://phabricator.services.mozilla.com/D82204
Changes:
- set up a new minimal docker image for `android-em` tests.
- reorganize commonly referenced files into a specific folder `taskcluster/docker/files` to reduce instances of same file being copy-pasted into directories.
- add android-test docker build task to taskcluster.
Differential Revision: https://phabricator.services.mozilla.com/D82204
Changes:
- in `mozharness_test.py`, add new optional key `job-script` similar to existing provision for toolchain tasks.
- set the default bootstrapping script depending on platform.
- substitute the correct script in the final mozharness call.
Differential Revision: https://phabricator.services.mozilla.com/D83559
Changes:
- create a `test-android.sh` script that is based on, but removes non-android related paths from the `test-linux.sh` script.
Differential Revision: https://phabricator.services.mozilla.com/D82252
Changes:
- set up a new minimal docker image for `android-em` tests.
- reorganize commonly referenced files into a specific folder `taskcluster/docker/files` to reduce instances of same file being copy-pasted into directories.
- add android-test docker build task to taskcluster.
Differential Revision: https://phabricator.services.mozilla.com/D82204
Disabled because we've observed that this is introducing noise on G5.
Also added perftuning to the other Pixel 2 tests.
Differential Revision: https://phabricator.services.mozilla.com/D83808
These scripts don't call `build-clang.py`, they just repackage artifacts from other tasks that do.
I went with `repack` over `repackage` since that seems to be the established pattern in `taskcluster/scripts/misc/`.
Differential Revision: https://phabricator.services.mozilla.com/D83532
This change doesn't seem to affect anything now, but when I was pulling out
some bits into `job-defaults`, the lack of a `job-name` resulted in some
extremely cryptic errors from inside taskcluster code.
Differential Revision: https://phabricator.services.mozilla.com/D83127
The explanatory comment at the top of the new file provides some context.
The larger motivating context here is wanting to use `job-defaults` to clean
up the toolchain fetches for the Linux build jobs. But if we did that, we
run into the problem that such fetches are merged with whatever the
individual job(s) specify for their toolchain fetches, not replaced. So
we'd wind up with multiple clang toolchains being downloaded, or similar,
and things would go downhill from there, as workers attempt to unpack those
things to the same location (which is bad) or perhaps error out (which is
also bad).
This change therefore paves the way for providing `job-defaults` in linux.yml.
Differential Revision: https://phabricator.services.mozilla.com/D83158
Currently the macosx-cross toolchain build pulls in a linux64-clang toolchain, uses it to build a Mac native toolchain, and then clobbers the result with pieces of the Linux toolchain. This means that the same version of LLVM is used to build the Mac pieces and be part of the final artifact. This will become a problem with upcoming LLVM 11 where we'll want to build the Mac pieces with LLVM 9 but otherwise repackage an LLVM 11 Linux toolchain.
So this commit makes the macosx-cross workflow look more like the win-cross one: take two compilers built elsewhere and just merge them.
Differential Revision: https://phabricator.services.mozilla.com/D83378
If we are generating only a part of the graph, to given kind, don't fail if a
build is packaging tests and there is no corresponding test task, as the tests
may not have been generated.
Differential Revision: https://phabricator.services.mozilla.com/D82097
LLVM 11 introduces a hard requirement for SDK 10.12 in order to build for Mac. We want to keep building older LLVMs with 10.11 though, so this patch adds some flexibility so that build-clang can make use of whatever SDK package a particular task pulls from tooltool (but still requesting a deployment target of 10.11).
Differential Revision: https://phabricator.services.mozilla.com/D82621
Changes:
- removes the ability to schedule web-platform-tests-reftest-backlog and web-platform-tests-backlog from non-opt variants of each platform.
- in addition, removes the ability to schedule the above from windows7-32, regardless of the variant.
- clean up references to now-deprecated platforms in backlog task definitions.
Differential Revision: https://phabricator.services.mozilla.com/D83172
We could make a new task for this, but `mozwebidlcodegen` depends on code in `mozbuild`, and vice-versa, so there doesn't really seem to be any meaningful advantage to that.
Differential Revision: https://phabricator.services.mozilla.com/D83187
We already have `value = source[key]` thanks to iteration; we should be
using `value` directly rather than re-doing `source[key]`.
Differential Revision: https://phabricator.services.mozilla.com/D83157
The `plain` builds are intended to make sure that building with an empty mozconfig doesn't break. Since an empty mozconfig build wouldn't have wasm sandboxing, neither should the `plain` builds.
While here, also remove wasi/lucet dependencies from the base-toolchain tasks, since they don't enable sandboxing either.
Differential Revision: https://phabricator.services.mozilla.com/D83011
The unit test is broken under Python 3 on macOS, so I haven't included any macOS version of this task; one should probably be added if/when that's fixed.
Differential Revision: https://phabricator.services.mozilla.com/D83169
Add an action that will trigger a task that runs
`mach release push-scriptworker-canary`
to test a new scriptworker deployment.
Differential Revision: https://phabricator.services.mozilla.com/D82821
Adds some additional assertions against what would be ran with |mach try auto|.
This also fixes a legit bug that the 'test_important_manifests_only' test
caught.
Differential Revision: https://phabricator.services.mozilla.com/D81404
This creates a general pytest plugin under "config/mozunit" which implements
the ability to mark tests as "slow". Slow tests will be skipped by default
unless "--run-slow" is passed in.
Differential Revision: https://phabricator.services.mozilla.com/D81402
This adds the support for backfilling of manifest scheduled tasks while
preserving the existing features (e.g. schedule a specific test path).
Remove temporary new-backfill action.
Differential Revision: https://phabricator.services.mozilla.com/D82599
This optional mode to specify a test path when backfilling has been broken for a while
and is not being used to make it worth fixing. It is a mode
originally designed to be used with the test-verify tasks.
Differential Revision: https://phabricator.services.mozilla.com/D82722
In Bug 1641971 (e7d4c490c31a) we changed `taskgraph.create.create_tasks` to
always get the decision task id to use. However, the call in create-interactive
passed `None`, expecting to have on generated. This changes that to use the
`create-interactive`s action task as the task id.
Differential Revision: https://phabricator.services.mozilla.com/D82652
This patch adds all desktop pageload tests to browsertime. It adds taskcluster tasks for these but doesn't enable them in production. They can only be run when --full is used.
Differential Revision: https://phabricator.services.mozilla.com/D82432
This patch adds an option to disable the firefox window recorder in browsertime, it also adds it to the browsertime-mobile taskcluster config since it is disabled there (but not on desktop).
Differential Revision: https://phabricator.services.mozilla.com/D82431
We were building a context with a top-level directory matching the image name
during graph generation. This is not accepted by docker build. This didn't
cause issues since we only used it for digest generation. Since we are
switching to use the decision generated context, change this to match the
context being generated when building an image.
Differential Revision: https://phabricator.services.mozilla.com/D77841
To support using kaniko[1] for building images, we need to generate the docker
contexts in a seprate task from the docker-image task. Since we use the hash of
the context as the cached-task digest, we generate the context in the decision
task already, so this adds a way to write that out to be used by downstream
tasks.
[1] https://github.com/GoogleContainerTools/kaniko
Differential Revision: https://phabricator.services.mozilla.com/D77839
I'm going to change this from being autodected, to specified explicitly, and so
`use-sccache` better captures the meaning.
Differential Revision: https://phabricator.services.mozilla.com/D82483
CLOSED TREE
Backed out changeset 93b955e5c048 (bug 1637544)
Backed out changeset be0717d76643 (bug 1637544)
Backed out changeset 447fea64b68d (bug 1637544)
I'm going to change this from being autodected, to specified explicitly, and so
`use-sccache` better captures the meaning.
Differential Revision: https://phabricator.services.mozilla.com/D82483
The current version hits OOM errors when dsymutil-ing files created by clang 10 and 11.
The choice of clang-10 here is somewhat arbitrary in that it's a fetch job that we conveniently already had in the tree. It doesn't have to be exactly version 10 specifically.
Differential Revision: https://phabricator.services.mozilla.com/D82453
Fixes the bustage described in 4e1faecce8
In theory I could leave this task alone and fork a new toolchain for use by clang 11, but this seems internal-enough that existing builds shouldn't be negatively affected by this.
Differential Revision: https://phabricator.services.mozilla.com/D82444
We already do this for e.g. `linux64-clang-9`; this patch extends that pattern everywhere.
This will make it easy to do try runs with other clangs: just move the `toolchain-alias` lines from the `9` tasks to the `trunk` tasks.
Also, this makes dependencies more explicit: for example the gn task specifically requests a clang-9 compiler, so it will also need a clang-9-based cctools-port, rather than whatever moving value the alias represents.
Differential Revision: https://phabricator.services.mozilla.com/D82441
Some cleanup before I add more copies of this task.
Since this is based on a repo called `cctools-port` it seems like it would be better to keep that substring intact.
Differential Revision: https://phabricator.services.mozilla.com/D82439
We use the symbol of the original task we backfill and suffix the revision of the push
from which we backfill. This makes it a bit more clear as to from which push we requested
the backfill.
We also modify the group symbol (if available) to group backfilled tasks.
***
***
Differential Revision: https://phabricator.services.mozilla.com/D81379
This backfill action schedules a secondary action per push. This secondary action
schedules the intended task on behalf of the primary action.
This introduces logic on how to trigger a hook by an action.
Differential Revision: https://phabricator.services.mozilla.com/D79351
This is the only place where we override the default of `required=False`.
Change it to match the rest of the schemas, so as not to confuse people who
don't happen to notice the final line of the schema.
Differential Revision: https://phabricator.services.mozilla.com/D81657
We don't ever set `depname` in any of the kinds that use these transforms.
Since we don't apply default values in schemas, `Required(..., default=...)`
behaves equivalently to `Optional(...)`, so we can just remove these options.
Differential Revision: https://phabricator.services.mozilla.com/D81654
This could go further and test for m-c or the trunk/integration aliases, etc. but I opted for not letting this increase try-syntax jobs.
Differential Revision: https://phabricator.services.mozilla.com/D81756
Right now setup_perftest_test_date adds --test-date yesterday to all perftest
runs. we want that only for the ones doing batches
Differential Revision: https://phabricator.services.mozilla.com/D81562
Mike Hommey
Joel Maher
sachin
Ben Hearsum
Gregory Mierzwinski
alexandru.ionescu
Florin Strugariu
Dorel Luca
Nathan Froyd
Marco Castelluccio
Justin Wood
Csoregi Natalia
Nick Thomas
Rail Aliiev
Andrew Halberstadt
Christian Holler
Tom Prince
Andrew Sutherland
Bogdan Tara
Johan Lorenzo
Brindusan Cristian
Miko Mynttinen
Andreea Pavel
Ricky Stewart
Geoff Brown
Andrew Creskey
David Major
Narcis Beleuzu
Butkovits Atila
Sebastian Hengst
Jeff Gilbert
Cosmin Sabou
Razvan Maries
Tarek Ziadé
Kartikaya Gupta
Mihai Alexandru Michis
Edwin Takahashi
Sylvestre Ledru
Kris Maglione
Mitchell Hentges
Armen Zambrano G.
Andi-Bogdan Postelnicu
Chris AtLee
emilfs
James Graham
Mihai Tabara
Simon Fraser
Martin Stransky
Henrik Skupin