This allows us to decouple GeckoView from exoplayer2, have it's own Java
settings and not pollute GeckoView's dependencies.
Differential Revision: https://phabricator.services.mozilla.com/D133792
This patch introduces a new field maven-artifacts which allows specifying a
list of maven artifacts that will be generated by a build and published to our
maven repository.
This allows us to split up GeckoView into separate maven packages.
Differential Revision: https://phabricator.services.mozilla.com/D133790
If that happens they'll all schedule `beetmover-geckoview` tasks that
write to the same bucket, and apparently make things in maven land go
sideways, so let's try to avoid it, like we already do in the
`nightly_desktop` case. There's still a race if two decision tasks run
at the same time, but it's much reduced.
Differential Revision: https://phabricator.services.mozilla.com/D135327
The test itself takes around 25 minutes. If the task hits the clone step, it will take an additional 10minutes.
Note that there is also a slow clone issue where clone can take between 15 and 30 mn.
Differential Revision: https://phabricator.services.mozilla.com/D135476
Ensure setpref=fission.autostart set appropriately for task fission-ness. Otherwise, verify that fission enabling/disabling is appropriate for the task.
Differential Revision: https://phabricator.services.mozilla.com/D134522
Ensure setpref=fission.autostart set appropriately for task fission-ness. Otherwise, verify that fission enabling/disabling is appropriate for the task.
Differential Revision: https://phabricator.services.mozilla.com/D134522
Rather than defining a distinct `Parameter` class for Gecko, this now uses the
`extend_parameters_schema` utility function (which mobile repos are already
using).
As a consequence, shared parameters are now defined in standalone taskgraph.
And only Gecko-specific parameters are listed in
`gecko_taskgraph/parameters.py`
The only exception is `project` which gets redefined so we can override the
standalone taskgraph default (since it derives `project` from the repo name,
which doesn't work for Gecko).
Differential Revision: https://phabricator.services.mozilla.com/D134515
pkgconf is an alternative implementation of pkg-config that is more
cross-platform. It has also become the default on Fedora, so it's not
some random project.
Differential Revision: https://phabricator.services.mozilla.com/D135009
pkgconf is an alternative implementation of pkg-config that is more
cross-platform. It has also become the default on Fedora, so it's not
some random project.
Differential Revision: https://phabricator.services.mozilla.com/D135009
This logic is Gecko specific and shouldn't be tied to the `Parameter`
class anyway. It makes more sense as a utility function.
Differential Revision: https://phabricator.services.mozilla.com/D134514
This logic is Gecko specific and shouldn't be tied to the `Parameter`
class anyway. It makes more sense as a utility function.
Differential Revision: https://phabricator.services.mozilla.com/D134513
Downloads disappear out of the /downloads/releases/ folder
when a newer version comes out, and they live forever in
/downloads/releases/archives/
Differential Revision: https://phabricator.services.mozilla.com/D132625
Downloads disappear out of the /downloads/releases/ folder
when a newer version comes out, and they live forever in
/downloads/releases/archives/
Differential Revision: https://phabricator.services.mozilla.com/D132625
We clean up our Gecko clone between tasks via `hg robustcheckout --purge`, which runs `hg purge`. This is very effective, *but* it doesn't detect or clean up any nested clones.
Because we run cross-channel on Gecko workers, and because we clone `comm/` in cross-channel and haven't cleaned it up, and because `hg purge` doesn't detect or clean up nested clones, and because our current virtualenv setup code traverses the tree and can error out on `comm/` clones, let's clean up `comm/` after running cross-channel.
We'll be moving TB cross-channel to different tasks/workers in bug 1742711, and ideally we can update robustcheckout and/or `hg purge` to be able to detect and/or clean up nested clones.
Differential Revision: https://phabricator.services.mozilla.com/D134582
This uses the same trick as bug 1743832 under the hood. We could go the
other way around, extracting the configure code to a separate module,
but the longer term goal here is to have configure figure out which
things to bootstrap for the selected build type.
As a side effect, mach bootstrap will stop re-bootstrapping things that
are already up-to-date, at least for things using
install_toolchain_artifact, excluding those that don't follow the
convention wrt the extracted directory path.
Differential Revision: https://phabricator.services.mozilla.com/D134595
This adds the libasound dependency on Linux which is required since we started
supporting Web MIDI. xpcshell has inherited this dependency and thus all
workers running it need to have it installed.
Differential Revision: https://phabricator.services.mozilla.com/D134600
Forward declarations of enums require a type specifier.
However, in the MinGW headers, the type specifier is missing.
MinGW can't just add it because this header comes from Wine
_and_ is auto-generated by widl. But widl doesn't support
adding the type specifier.
There's an in-progress patch at
https://github.com/rbernon/wine/compare/wip/upstream...wip/widl/v1.patch
but I can't wait for that to land as the author indicated it's
not being actively pursued.
Differential Revision: https://phabricator.services.mozilla.com/D134235
This is required to build the midir crate on Linux. The Dockerfile change is
needed to run the just built xpcshell executable on the build host.
Differential Revision: https://phabricator.services.mozilla.com/D124641
Also update the table of Firefox requirements. It's worth noting that I
actually upgraded to 1.57 earlier than I should have (too close to
freeze), but what is done is done.
Differential Revision: https://phabricator.services.mozilla.com/D134124
This attribute isn't necessary as e10s is the default. Instead consumers can
check if the string '1proc' appears in the unittest_variant attribute.
Depends on D133225
Differential Revision: https://phabricator.services.mozilla.com/D133226
Tasks that set this key to false *will not* run under the standard no variant
configuration.
This is needed to replace the 'e10s=false' case (i.e tasks that *only* run with
1proc like mochitest-chrome or mochitest-a11y.
Depends on D133224
Differential Revision: https://phabricator.services.mozilla.com/D133225
Previously we we're validating the entire 'test_description_schema' again, even
though the vast majority of those keys were in fact no longer needed. This was
preventing us from ever removing keys from the 'task' object. Which I believe
is a good practice to keep things simple.
Differential Revision: https://phabricator.services.mozilla.com/D133222
We separate the patchset from the per-platform configuration, which
will ensure we keep the same patchset across them (spoiler alert: we
weren't).
Also, as most builds are PGO, use that in per-platform configurations,
and add an override config that sets it back to 2-stages for the
builds we don't want to PGO.
Differential Revision: https://phabricator.services.mozilla.com/D133317
I forgot that mozboot only pulls in updates when you run `./mach bootstrap`,
so some people got the new mozcrash.py locally without actually having the
new rust-minidump-based version. So now we first run the stackwalk binary
with -V to check what version it is.
The rest of the details can be found in the comments I added.
Also updates rust-minidump to 0.9.6 get some CLI parsing fixes
and better --help documentation (socorro staging is already updated to
this version).
Differential Revision: https://phabricator.services.mozilla.com/D133251
99% of the fetch tasks that pull from a git repo pull from a specific
revision, as specifically intended. There is however one exception,
which is the clang-trunk fetch, which is intended to always pull the
last revision on clang trunk. Unfortunately, the log also doesn't say
what specific commit it checked out.
Ironically, the log for all the other tasks does say which commit is
checked out, because git prints it out when it goes in detached head
state, which it does when checking out a specific sha1 (so in that
case, it's printing something we already know).
In the one instance where we checkout a branch, we don't rely on the
branch being checked out, so we just forcibly go in the same state of
detached head in that case too.
Differential Revision: https://phabricator.services.mozilla.com/D133063
Now that `use_fetches` runs after `make_task_description`, tasks that
come in look more like tasks from kind_dependencies, and the code to
extract the alias, artifact and environment info can now be shared.
Differential Revision: https://phabricator.services.mozilla.com/D132650
This will help to avoid the duplication of the code that takes artifact
names, environment and aliases for each fetch, and will allow changes to
toolchain.py that wouldn't be possible otherwise.
The only difference it makes to the resulting taskgraph is that scopes
are not in the same order as before, but that doesn't matter.
Differential Revision: https://phabricator.services.mozilla.com/D132649
This is being split to a new file mainly because the worker definitions take up
a lot of LOC. Getting this out of the main transforms makes it easier to
navigate.
Differential Revision: https://phabricator.services.mozilla.com/D132411
Chunking transforms should always go last since there should never be
configuration that is different between chunks (other than what is set by the
chunking transforms themselves).
Running them last also ensures we don't do extra work as the number of tasks
grows by 10x or more after splitting chunks.
Differential Revision: https://phabricator.services.mozilla.com/D132410
Future commits will tease this apart a bit more, but for now this helps
crystallize the order in which transforms are applied. The flow of the overall
test transforms goes something like this:
1. Enter 'transforms/test/__init__.py'
2. Validate all tasks against the test_description_schema
3. Run sibling transforms (starting with 'variant.py' and ending with 'other.py' for now)
4. Make the job description
As we pull more transforms out of 'other.py' and into their own smaller
transform files, it will be clear that the order in which these smaller files
run is important. Adding new transforms will no longer involve picking some
random spot to insert it.
Differential Revision: https://phabricator.services.mozilla.com/D132409
Many things need to be set "by variant", so therefore we need to resolve them
early on in the process. By pulling them out into a separate file that
explicitly runs before the "__init__.py" transforms, we A) make it harder
for someone to accidentally add their transform ahead of them, and B) reduce
some of the clutter in "__init__.py".
Differential Revision: https://phabricator.services.mozilla.com/D132408
Bug 1713258 added KEY to the beetmover manifest, but it gets excluded
from `upstreamArtifacts` in `generate_beetmover_upstream_artifacts`
because it's not in the signing task's `release-artifacts` attribute,
and so it doesn't actually work.
Differential Revision: https://phabricator.services.mozilla.com/D132470
This patch changes the viewport retries to 60. This means that we will now check the first 1 second of the video to try to find a viewport.
Differential Revision: https://phabricator.services.mozilla.com/D132379
This patch adds a new toolchain run property: toolchain-env which allows
specifying additional environment variables that will be added to the worker
when consuming the toolchain artifact.
Differential Revision: https://phabricator.services.mozilla.com/D131987
We have a 3 way circular dependency in the test transforms:
1. The 'raptor' transforms depend on 'split_variants'
2. The 'get_mobile_project' utility function depends on the 'raptor' transforms
3. The 'split_variants' transform depends on the 'get_mobile_project' utility function
The only reason things are working is that in step 3 we don't need the result
of 'get_mobile_project' to be perfect. This revision breaks the circular
dependency by checking for 'android' in the test-platform rather than relying
on 'get_mobile_project == geckoview'.
While this is admittedly not as precise, we only apply the particular variant
that needs this to geckoview tasks in the first place. In other words, this
revision does not cause any additional tasks to be added / removed.
Differential Revision: https://phabricator.services.mozilla.com/D132072
We'd like to break down the test transforms into smaller well-ordered pieces.
This will allow us to run any transforms defined in files under the
'transforms.test' directory.
Differential Revision: https://phabricator.services.mozilla.com/D132071
This ensures validation happens before we handle variants and raptor. It means
we no longer make any meaningful changes to the graph before validating.
Differential Revision: https://phabricator.services.mozilla.com/D132070
In the test transforms we currently have general 'test_description_schema' and
a schema for raptor / browsertime tasks. Prior, the raptor schema was validated
*before* the test schema, and we were doing a lot of validation twice.
This revision switches the order such that the test schema is first evaluated,
and then the raptor one is. To accomplish this, we create a 'raptor' subconfig
that the 'test_description_schema' completely ignores (and leaves up to the
raptor schema to evaluate).
The benefit of this change is that all test tasks will have a single entry point
when they get into the 'test' transforms. This makes reasoning about their
configuration much easier and lays the ground work to start splitting even more
transforms out into other modules.
Differential Revision: https://phabricator.services.mozilla.com/D132069
This includes:
transforms/tests.py -> transforms/test/__init__.py
transforms/raptor.py -> transforms/test/raptor.py
This is a pre-cursor to splitting the file up into multiple smaller files under
the new 'test' transform directory.
Differential Revision: https://phabricator.services.mozilla.com/D132068
Since we're going to enable EFT on Nightly and let it bake there a bit to
get feedback, we might face a situation where we have to disable EFT.
Having these test variants will help us to not regress non-EFT cases so if
we ever have to rollback EFT, there wouldn't be extra work.
Depends on D131477
Differential Revision: https://phabricator.services.mozilla.com/D131600
Since we're going to enable EFT on Nightly and let it bake there a bit to
get feedback, we might face a situation where we have to disable EFT.
Having these test variants will help us to not regress non-EFT cases so if
we ever have to rollback EFT, there wouldn't be extra work.
Depends on D131477
Differential Revision: https://phabricator.services.mozilla.com/D131600
Consolidate Mach virtualenv management to the front of the
Mach process. This obsoletes `./mach create-mach-environment`
and simplifies the `sh` portion of the top-level `./mach` script.
This helps ensure that the Mach virtualenv doesn't become
out-of-sync and simplifies the mental model of the Mach
virtualenv situation.
Differential Revision: https://phabricator.services.mozilla.com/D120401
`--no-virtualenv` was needed for one use case: the
`get_and_diffoscope` task, despite not needing `psutil`, would
run into failures during virtualenv-creation because we used to
unconditionally build the `psutil` package, and `get_and_diffoscope`
didn't have the environment needed for such a build.
Since we no longer build and install `psutil` into every virtualenv,
it's no longer needed for its one usage, which means that it can be
removed.
`--requirements` is replaced by `--virtualenv`, which removes an
ad-hoc pip package installation and embraces the centralized dep
system.
`--no-activate` is now implied by default: a virtualenv is only
created and activated if `--virtualenv` is provided.`
`ipython==7.16.1` was the chosen version because it is the last
one compatible with Python 3.6.
Differential Revision: https://phabricator.services.mozilla.com/D131529
Since we're going to enable EFT on Nightly and let it bake there a bit to
get feedback, we might face a situation where we have to disable EFT.
Having these test variants will help us to not regress non-EFT cases so if
we ever have to rollback EFT, there wouldn't be extra work.
Differential Revision: https://phabricator.services.mozilla.com/D131600
This is the first step in replacing a huge pile of our breakpad-based infra
with our new implementation (rust-minidump). This stackwalker is only used
for reporting crashes in local builds and CI, so it's a good first deploy.
Although most of the work on rust-minidump has been focused on the JSON output,
this uses the --human output, because it's primarily intended for humans to
directly read. There is however some minor parsing done on this format. This
is not *strictly* supported by --human (it has no schema) but it's not something
we plan to break. (This parsing is pre-existing, just recording the facts.)
The new build configs/scripts are hybridized from fix-stacks and dump_syms,
as this basically is a hybrid of the two. In particular it needs the openssl
vendoring tricks that dump_syms uses, but is a target binary that prefers
win32 over win64 (like fix-stacks).
Technically a regression but probably just culling legacy cruft at this point:
this patchset removes support for building a local copy of minidump-stackwalk
from source. You must now download a copy built on task-cluster using mozboot.
mozboot *already* did this, which is why this feature appears to be legacy cruft
-- there was little reason to build a local copy.
However rust-minidump's minidump-stackwalk has a far better portability story,
so you can build+install your own local copy by just running:
cargo install minidump-stackwalk
Differential Revision: https://phabricator.services.mozilla.com/D131315
This will make it easy for consumers to check whether a set of tasks are on the
same test setting or not by comparing their hash.
This also uses a 'ReadOnlyDict' to ensure the setting isn't modified later.
Differential Revision: https://phabricator.services.mozilla.com/D131285
This avoids re-parsing the build/test platform a second time by re-using what
is already defined in the test setting.
This does result in a few differences in the taskgraph, but in all cases that
I've been able to detect, it's actually *fixing* errors that were previously
going uncaught.
Differential Revision: https://phabricator.services.mozilla.com/D131283
This also adds way to bypass the 'check_schema' call since this schema's keys
contains strings from `variants.yml` which sometimes have underscoes in them.
Differential Revision: https://phabricator.services.mozilla.com/D131282
This will:
1. Give consumers of test tasks a way to reliable determine the configuration
it is running under without needing to parse magic labels.
2. Consolidate much of the logic around parsing the build and test platform
strings.
Differential Revision: https://phabricator.services.mozilla.com/D131281
This was dumping the entire output of the generation *after* the pytest
failure. Which basically makes these tests impossible to work with (as you need
to scroll up forever to find the error). If you had a previous test run in
your shell buffer prior, it was really difficult to find the start of the
current run.
Ideally we'd still have a way of seeing this ouput when requested.. maybe we
could save it to a file? But for now this is a *much* saner default. Plus if
additional context is needed, we can focus on re-writing the tests to provide
it in their error messages.
Differential Revision: https://phabricator.services.mozilla.com/D131280
This patch adds 4 new interactive tests for facebook and reddit. The patch updates the recordings as well as needed. Furthermore, an update to the visualmetrics.py script is needed to handle a permafailure in the reddit-post tests. Note that the reddit-post test was split in 2 since they are quite large.
Differential Revision: https://phabricator.services.mozilla.com/D130665
These logs have been added to help with the investigation of shutdown issues. But it turned out to not be that helpful. Lets get it removed from wdspec jobs to lesser clutter the logs.
Differential Revision: https://phabricator.services.mozilla.com/D130974
This replaces the following variants with composites:
webrender-sw-a11y-checks
webrender-sw-fission
webrender-sw-wayland
The full task graph is identical after this change.
Differential Revision: https://phabricator.services.mozilla.com/D130400
Often we have two variants that we want to test together. The only way to do
this currently is to create a 3rd variant that is the sum of the previous two.
This feature will allow tasks to define multiple variants using the '+' token.
E.g:
variants: ["fission", "webgl-ipc", "fission+webgl-ipc"]
The third variant there will first apply the 'fission' configuration, then the
'webgl-ipc' configuration (in order).
Differential Revision: https://phabricator.services.mozilla.com/D130399
Listing contacts next to the variant will provide a point of contact in case
there are questions around cost / capacity or relative importance of a given
variant.
Differential Revision: https://phabricator.services.mozilla.com/D129834
This also splits the blurb for the 'test' kind into a new file as it was
getting too long for kinds.rst. Having documentation for every kind in the same
file tends to lead towards uninformative documentation.
Differential Revision: https://phabricator.services.mozilla.com/D129830
This ensures the filters stay alongside the variant definitions. I called the
key 'when' as 'filter' is ambiguous about whether the tasks that match are
being included or excluded.
Differential Revision: https://phabricator.services.mozilla.com/D129828
Listing contacts next to the variant will provide a point of contact in case
there are questions around cost / capacity or relative importance of a given
variant.
Differential Revision: https://phabricator.services.mozilla.com/D129834
This also splits the blurb for the 'test' kind into a new file as it was
getting too long for kinds.rst. Having documentation for every kind in the same
file tends to lead towards uninformative documentation.
Differential Revision: https://phabricator.services.mozilla.com/D129830
This ensures the filters stay alongside the variant definitions. I called the
key 'when' as 'filter' is ambiguous about whether the tasks that match are
being included or excluded.
Differential Revision: https://phabricator.services.mozilla.com/D129828
The fact that the test runner app is defined inside the geckoview test package
has always felt like a hack to me. I've mistakenly thought that
TestRunnerActivity was used in GeckoView's junit tests many times (even though
that's not the case).
From what I can see, there's no way to generate an AAB package for androidTest,
so to be able to run Gecko tests as AAB we finally need to define the
TestRunner as an ordinary package instead.
Differential Revision: https://phabricator.services.mozilla.com/D127320
This tool is used to install AAB packages to devices.
AAB is the new package format for Android Apps and will supersede APK.
Differential Revision: https://phabricator.services.mozilla.com/D127318
This also means we can remove the 'handle_keyed_by_variant' transform as this
was only needed to resolve things that had 'by-variant' and were deferred in
the regular 'handle_keyed_by' transform. Since variants are now handled early,
we no longer need to defer them here.
Differential Revision: https://phabricator.services.mozilla.com/D129431
Changed wget output log granularity from 📣 to :giga: (only used in case the python library didn't cover all usecases)
Added python methods to async cache/download all required files for update verify
Modified Dockerfile requirements to include aiohttp python lib
Differential Revision: https://phabricator.services.mozilla.com/D129429
I forgot to update the module name in the new '--diff' feature when renaming
the module to 'gecko_taskgraph'.
This syncs the commit to standalone taskgraph here:
c3b465290f
Differential Revision: https://phabricator.services.mozilla.com/D129558
esr91 should now update the regular (non-next) ESR bouncer aliases, and
update-verify should test updates from esr 68.x to 91.x.
Differential Revision: https://phabricator.services.mozilla.com/D129397
The only non-github CI change is the addition of `default-features =
false` to the error-chain dependency, which removes the need for e.g.
the object crate.
Differential Revision: https://phabricator.services.mozilla.com/D129113
Jesse Schwartzentruber
Agi Sferro
Joel Maher
Christian Holler
Iulian Moraru
Kershaw Chang
Julien Cristau
Julian Descottes
andrej
Mike Hommey
Geoff Brown
Butkovits Atila
Ryan VanderMeulen
Tom Ritter
Noemi Erli
Alexandru Michis
Andrew Halberstadt
Sebastian Hengst
Narcis Beleuzu
Yury Delendik
Aki Sasaki
Michelle Goossens
Gabriele Svelto
Gregory Mierzwinski
Alexis Beingessner
Barret Rennie
Heitor Neiva
Sandor Molnar
Sylvestre Ledru
Marco Castelluccio
Nicolas Chevobbe
Cosmin Sabou
Mitchell Hentges
Cristian Tuns
Marian-Vasile Laza
Henrik Skupin
Andi-Bogdan Postelnicu
Norisz Fay
Kimberly Sereduck
Csoregi Natalia
Connor Sheehan