Starting with Firefox 84 when macOS switched to run its tests with WebRender
(bug 1673071), the wdspec and wpt crashtests also ran in the DevEdition
configuration because there were rules ".*-qr/.*: ['release']".
Differential Revision: https://phabricator.services.mozilla.com/D103283
This patch adds a new "--ci" argument for the
"mach puppeteer-test" command. As such it can
also be used locally to simulate a test job in CI.
Differential Revision: https://phabricator.services.mozilla.com/D101780
Bug 1553230 made configure automatically get toolchain artifacts in some
cases. The artifacts for clang-cl builds are clang.tar.zst, and extract to
clang/. Configure derives the task name from that knowledge, and fails
to find clang-cl tasks because of that.
For consistency, these tasks should be called clang. They are clang
builds anyways, and like any other clang builds, they also contain
clang-cl.
Differential Revision: https://phabricator.services.mozilla.com/D103150
For simplicity, this implements just on in `NO_TASKS` (the default) or
on in `ALL_TASKS` (opt-in). This disables all category registrations
when in background task mode; we'll selectively re-enable things as
appropriate.
The flag constants were chosen to smoothly extend to a (16-)bit set in
the future, should we want to add a `JUST_TASKS("task", "other-task")`
option in the future.
This also adds ython tests for gen_static_components.py exercising
categories, simply 'cuz it's easiest to see what this adds in such
tests. Functional tests will follow in patches that actually
implement the new background tasks functionality.
Differential Revision: https://phabricator.services.mozilla.com/D96654
Per bug 1686507 comment 5, tier1 things should run on at least autoland and try. These tasks have too many dependencies that would need to start running on autoland, so I think it's not worth the trouble and I'd rather put them back to tier-2.
Differential Revision: https://phabricator.services.mozilla.com/D102336
This patch changes the formatting of the error lines output by the vismet tasks so they can be properly picked up by the Treeherder parser.
Differential Revision: https://phabricator.services.mozilla.com/D102763
remoteautomation.py is an old collection of code used by android mochitest and android reftest;
it survived the removal of automation.py. This patch removes remoteautomation.py, moving the
majority of the functionality to a new class in mozdevice. Some features are simplified or
removed, and the remainder moved into the remote mochitest/reftest harnesses.
Differential Revision: https://phabricator.services.mozilla.com/D102239
Now that we use an external dump_syms, we don't need to build
breakpad's.
This means we also don't need the dump_syms_rust_demangle crate anymore.
Differential Revision: https://phabricator.services.mozilla.com/D101865
The previous blockers for having this task are no longer an issue, as far as I can tell. The only perma-failure is bug 1612035. Until that's addressed we can remove those files with `pre-diff-commands` and have something rather than nothing.
Differential Revision: https://phabricator.services.mozilla.com/D101630
This patch makes it so that any retrigger actions being performed from Treeherder on browsertime tasks (through add-new-jobs, retrigger, and retrigger-multiple) will always retrigger the vismet (visual-metrics) tasks correctly. Furthermore, when a vismet task is retriggered, we'll retrigger the browsertime test task.
Differential Revision: https://phabricator.services.mozilla.com/D101404
rename_gcov_flush_clang_11.patch doesn't apply cleanly to clang 12 and will need to be forked. Turning the filename into a parameter will let us have the tasks exist side by side.
Differential Revision: https://phabricator.services.mozilla.com/D101495
Unlike the binary diffs, the generated-files diffs have been stable for a long time. If these tasks ever start failing, it would be good to learn about that at tier-1 speed.
Differential Revision: https://phabricator.services.mozilla.com/D101629
The BaseApp has been renamed to follow org.mozilla.firefox ID.
Additionally update the Docker image used to run the flatpak build.
Differential Revision: https://phabricator.services.mozilla.com/D100647
Passing all the directories to mach allows to take more advantage of
parallelism (plus not having to start up mach and the whole export once
for every directory, which itself takes a bit of time)
This takes:
GECKO_PATH=. taskcluster/scripts/misc/source-test-clang-non-unified.sh
from about ten mins to under two minutes on my machine.
Differential Revision: https://phabricator.services.mozilla.com/D101320
Now that linux64-clang-11-macosx-cross doesn't need a macosx64-clang-*
to build, there is no circular dependency to avoid.
Differential Revision: https://phabricator.services.mozilla.com/D101080
The new toolchain contains both aarch64 and x86_64 compiler-rts. We
could have a separate for each, but compiler-rt is small enough that
entirely separate toolchains is not warranted.
Differential Revision: https://phabricator.services.mozilla.com/D101079
The llvm build system does support building a universal compiler-rt for
multiple platforms at once, but as far as I know it only supports doing
so with the same SDK, while we want to use separate SDKs for each, so
build the x86_64 compiler-rt separately.
Differential Revision: https://phabricator.services.mozilla.com/D101078
We actually don't need the macosx-cross toolchain to build compiler-rt,
because building compiler-rt doesn't require compiler-rt (duh).
Differential Revision: https://phabricator.services.mozilla.com/D101077
The webrender wrench macos build, which is cross-compiled, needs a
macOS native libLLVM (a .dylib) to link against. The file is currently
part of the macosx-cross toolchain, but that was more incidental than
intentional. As we're going to change the macosx-cross toolchain in a
way that will remove the libLLVM.dylib, pull the file from the macOS
native clang.
Differential Revision: https://phabricator.services.mozilla.com/D101150
Only sanitizer builds require a native llvm-symbolizer executable.
Ideally, we'd build llvm-symbolizer from scratch, which would be faster,
but for now, let's go the easy route and just extract it from the
corresponding native clang builds.
We don't actually do anything with the llvm-symbolizer executable on
android builds, so we don't install it in $FINAL_TARGET, avoilding
the dependency on android builds (plus, we actually don't have an
android-native llvm-symbolizer, so even if it were already shipped, it
would be the wrong file).
Differential Revision: https://phabricator.services.mozilla.com/D101076
Now that linux64-clang-11-macosx-cross doesn't need a macosx64-clang-*
to build, there is no circular dependency to avoid.
Differential Revision: https://phabricator.services.mozilla.com/D101080
The new toolchain contains both aarch64 and x86_64 compiler-rts. We
could have a separate for each, but compiler-rt is small enough that
entirely separate toolchains is not warranted.
Differential Revision: https://phabricator.services.mozilla.com/D101079
The llvm build system does support building a universal compiler-rt for
multiple platforms at once, but as far as I know it only supports doing
so with the same SDK, while we want to use separate SDKs for each, so
build the x86_64 compiler-rt separately.
Differential Revision: https://phabricator.services.mozilla.com/D101078
We actually don't need the macosx-cross toolchain to build compiler-rt,
because building compiler-rt doesn't require compiler-rt (duh).
Differential Revision: https://phabricator.services.mozilla.com/D101077
Only sanitizer builds require a native llvm-symbolizer executable.
Ideally, we'd build llvm-symbolizer from scratch, which would be faster,
but for now, let's go the easy route and just extract it from the
corresponding native clang builds.
We don't actually do anything with the llvm-symbolizer executable on
android builds, so we don't install it in $FINAL_TARGET, avoilding
the dependency on android builds (plus, we actually don't have an
android-native llvm-symbolizer, so even if it were already shipped, it
would be the wrong file).
Differential Revision: https://phabricator.services.mozilla.com/D101076
In addition to the usual dot-release type of fixes, this also lets us drop a good amount of code that we had patched into our clang 11.
Differential Revision: https://phabricator.services.mozilla.com/D100959
Bug 1683824 updated to the 0.2.14 tag, but that's patched in order to
publish to crates.io such that tiny-http is not patched anymore.
Unfortunately, that breaks sccache-dist, which is a known issue and will
eventually fixed when addressing
https://github.com/mozilla/sccache/issues/912.
So here we update to a corresponding version on master (the first that
bumps the version to the next number).
Differential Revision: https://phabricator.services.mozilla.com/D100863
I was waiting for a better reason to do this, because the cbindgen
changes from 0.15.0 to 0.16.0 don't break trunk builds. But since
downstream has updated (see bug 1684180) and there's no reason not to,
let's do this to avoid future churn.
Differential Revision: https://phabricator.services.mozilla.com/D100499
Since e10s-multi is going to be enabled by default, this gives us the ability
to run junit in a secondary, e10s-single configuration so that we can see them
both side-by-side for a little while, at least until we're settled in with the
e10s-multi configuration.
Differential Revision: https://phabricator.services.mozilla.com/D94883
When building compiler-rt for macOS, its build system assumes the
compiler used is a native macOS clang, drops all the cmake C/C++ flags,
and adds its own.
Because we do need to pass `-target $target` (otherwise we end up with
ELF x86_64 objects) and `-mcpu=apple-a12` for the correct baseline for
arm64 macOS, and because there is unfortunately no cmake variable that
the compiler-rt build system will use, and because CMAKE_C*_COMPILER
need to be a program without arguments, we need to wrap the compiler.
While here, add `-v` to the ninja call to have more useful logs.
Differential Revision: https://phabricator.services.mozilla.com/D99841
This patch enables ASan for rust code in the `linux64-asan/{debug,opt}` and `linux64-asan-fuzzing` build tasks. Since it is a `-Z` unstable flag, we'll move to the same `linux64-rust-nightly` toolchain that TSan builds use.
For the time being, the ASan Nightly Reporter builds are left alone, until we're more familiar with the perf impacts of this landing.
Differential Revision: https://phabricator.services.mozilla.com/D99556
At this point it's pretty clear that we won't be reverting to clang-9.
This doesn't remove everything with clang-9 in the name. I did a mark-and-sweep GC by hand so this only removes unused entries. Some are still active, e.g. linux64-clang-9 is used to build a number of misc helper tools.
Differential Revision: https://phabricator.services.mozilla.com/D99721
This requires tests to specify `python-3: true` in order to be run
with Python 3. When nothing is specified things work just like today,
so it's a more conservative change than the one in bug 1672181.
Obviously in the long term we will remove this and switch to Python 3
only, but this unblocks moving harnesses to py3 today.
Differential Revision: https://phabricator.services.mozilla.com/D98766
Instantiating a wasm library duplicates a file descriptor for /dev/null 3 times to be used as input, output and error streams for the wasm sandboxed code. When a lot of sandboxes are created and destroyed, a lot of descriptors are duplicated and closed. While this should be fine, POSIX does not seem to happy with the opening and closing of many file descriptors --- this could perhaps be some strange interaction with Firefox's seccomp filters and cross-process file descriptor handling as it is difficult to repro this outside of firefox.
However, the simpler fix here was to just eliminate the duplication of /dev/null and return an error when input, output or error streams are accessed by wasm sandboxed code. This means calls to printf will fail, but no code I know off actually checks the int error code returned by printf and this change is certainly compatible with existing sandboxed components.
Differential Revision: https://phabricator.services.mozilla.com/D99160
This requires tests to specify `python-3: true` in order to be run
with Python 3. When nothing is specified things work just like today,
so it's a more conservative change than the one in bug 1672181.
Obviously in the long term we will remove this and switch to Python 3
only, but this unblocks moving harnesses to py3 today.
Differential Revision: https://phabricator.services.mozilla.com/D98766
This is step 1 to make the docs here better, as discussed. Another followup
will come in a day or two to update the text/links of the docs to refer to
this and the latest version of repack-rust.
Differential Revision: https://phabricator.services.mozilla.com/D99128
We ended up with a symbol `mda2-dafa26b89ed-bk-1754415059f-bk`, which is longer
than the allowed schema length of 25 characters. Let's keep the existing
backfill suffix per sheriffs.
Differential Revision: https://phabricator.services.mozilla.com/D98694
Triggered by the platform name change in bug 1673071 when macOS switched to
running its tests with WebRender enabled.
Differential Revision: https://phabricator.services.mozilla.com/D99095
clang 12 (specifically https://reviews.llvm.org/D91379) made some refactorings to libc++ that exposed a problem in the MinGW headers. That has now been fixed upstream.
In the meantime the headers also gained definitions for ProcessPayloadRestrictionPolicy, so we can remove our workaround for that.
Differential Revision: https://phabricator.services.mozilla.com/D98945
To build native python packages, the python dev packages are needed.
This should resolve the psutil installation failure.
Differential Revision: https://phabricator.services.mozilla.com/D98197
Removes a bunch of zstandard installations that were ran from
within the mach virtualenv, which should already have zstandard
installed.
Differential Revision: https://phabricator.services.mozilla.com/D98387
To ensure l10n updates are still picked up by beta builds in a timely
fashion, remove "DONTBUILD" from commit messages when running on
mozilla-beta, and run a couple of hours before the "daily-releases" job
starts.
Differential Revision: https://phabricator.services.mozilla.com/D98349
We keep dump-syms on 1.47 because it breaks at runtime when built with
1.48. It also puts it in sync with other toolchains that won't upgrade
until we need them to.
Differential Revision: https://phabricator.services.mozilla.com/D98422
The zstandard package is always installed in the mach virtualenv.
The patch assumes that zstandard is only used from the mach virtualenv,
and never the build virtualenv.
Differential Revision: https://phabricator.services.mozilla.com/D98387
While all toolkit and js-based projects make use of mfbt, some others,
like tools/crashreporter and tools/update-packaging, don't.
So instead of including mfbt from the top-level directory, include it
from the relevant project top-level mozbuilds.
This allows to remove the dependency on mfbt files in the hash for the
minidump-stackwalk and mar-tools toolchains.
Differential Revision: https://phabricator.services.mozilla.com/D98378
To ensure l10n updates are still picked up by beta builds in a timely
fashion, remove "DONTBUILD" from commit messages when running on
mozilla-beta, and run a couple of hours before the "daily-releases" job
starts.
Differential Revision: https://phabricator.services.mozilla.com/D98349
Bug 1672888 set up a plain opt build that runs on m-c only, like the
other plain opt builds. It didn't add plain debug builds on integration
branches as well to avoid having too many jobs running on the slim
mac worker pool, but since bug 1678154 moved most jobs off them, we can
now add those.
Differential Revision: https://phabricator.services.mozilla.com/D98279
Install `psutil` when setting up the `mach` `virtualenv`s and stop importing the in-tree version in the build.
Nothing in-tree currently assumes or mandates the installation of `psutil` (all uses of `psutil` are guarded with imports of the form `try : import psutil; except ImportError: psutil = None`), so there's no back-incompatibility concerns here. There will be an awkward period where telemetry will be lacking CPU/disk data for everyone until they re-run `mach bootstrap` or `mach create-mach-environment`, but that will come back as people gradually update their `virtualenv`s.
An alternative to circumvent that issue is REQUIRING that `psutil` be installed by adding an assertion in `mach` that `psutil` can be found (allowing us to remove all the conditional logic in-tree around whether `psutil` is installed), but I wouldn't claim that we're ready to do that and deal with whatever fallout might occur.
Differential Revision: https://phabricator.services.mozilla.com/D90914
To avoid breakage from Ubuntu package upgrades, we split the test image
into two, one that sets up the packages only, and that won't change when
we need to update our scripts, and another that derives from it, that
adds all our scripts and other setup.
Additionally, we work around the recent timeout issues due to the
upgrade of packages.
The timeout itself is due to gst-launch waiting indefinitely when it
crashes, rather than exiting with an error code. Bug 1679491 addresses
this issue, but the core problem is that gst-launch crashes, which seems
to be that some change in libc (presumably "Fix pthread_rwlock_try*lock
stalls") turns `gst_object_unref: assertion '((GObject *)
object)->ref_count > 0' failed` fatal warnings (which were already
happening) into actual crashes (presumably because a race condition is
lost on a use-after-free).
This workaround, however, will stop working as soon as the updated libc
package migrates from bionic-updates into bionic proper, presumaby on
the next 18.04 dot-release. Hopefully, we won't be rebuilding the base
image for a while, avoiding further problems. Eventually, we'll want to
upload the base image to docker hub so that it's set in stone, and
change the FROM in the base image to use that instead.
Differential Revision: https://phabricator.services.mozilla.com/D98045
This adds a linux64-rust-dev toolchain and a git fetch of rust-lang/rust
that it can source from (stable tag for 1.47).
There are some issues with cross-compiling, so for now the toolchain only
builds a host build, although a lot of the machinery for cross compiling is
there for anyone brave/desperate enough to get it working.
Also note some changes were made to Rust's config.toml between 1.47 and 1.50,
so some version detection may need to be added in the future.
There is experimental support for providing patches via a new --patch flag.
Additionally, I documented the existence of the "bors-" mode.
Differential Revision: https://phabricator.services.mozilla.com/D97497
And don't set SHELL on mac workers (added in da452e43b5d5 because of the
exception thrown by this code not having a fallback)
Differential Revision: https://phabricator.services.mozilla.com/D97686
Backfill tasks might run with a slightly different definition than the task
with the same label in the current task graph. When we encounter them, ensure
we create a new task with an (almost) identical definition, rather than
grabbing it from the current push's full-task-graph.json.
There's already an action that has this behaviour (that is used for the
decision task). So rather than re-implement this behaviour in the regular
retrigger action, this patch ensures that the 'retrigger-decision' action
applies to backfills rather than the regular one.
Differential Revision: https://phabricator.services.mozilla.com/D97352
According to the documentation, actions are supposed to have an "order" system,
where the first action that applies to a task via its context, is supposed to
take precedence. But when running the match algorithm, we never break out of
the loop. So we end up applying the *last* action that matches a tasks' context.
As far as I can tell, this means we have been using the 'retrigger-disabled'
action for everything and things like retriggering Decision tasks don't work.
Differential Revision: https://phabricator.services.mozilla.com/D97351
If we aren't able to assign the appropriate manifests to the backfill, the
backfill is useless. Let's just let the exception propagate and fail loudly,
rather than log an error and keep going.
Differential Revision: https://phabricator.services.mozilla.com/D97350
remove default scheduling of talos, raptor, awsy on windows7-32. Ensure these are still available on try server. There is one exception, the talos-webgl-gli job, this is scheduled via a transform and I would rather leave that alone or remove all gli tests.
Differential Revision: https://phabricator.services.mozilla.com/D97310
Sebastian Hengst
Henrik Skupin
Andreea Pavel
Mike Hommey
Dorel Luca
Aki Sasaki
Gregory Mierzwinski
Joel Maher
Michael Kaply
Nick Alexander
Marco Castelluccio
David Major
Alexis Beingessner
Geoff Brown
Yura Zenevich
Mihai Tabara
Noemi Erli
Bartłomiej Piotrowski
Dan Mosedale
Alex Ionescu
Emilio Cobos Álvarez
Andi-Bogdan Postelnicu
Razvan Maries
Ben Hearsum
Florin Strugariu
Gabriele Svelto
Jeff Muizelaar
ffxbld
Cosmin Sabou
Aaron Klotz
Andrew Erickson
Tarek Ziadé
Tom Ritter
James Graham
Jamie Nicol
shravanrn@gmail.com
Mihai Alexandru Michis
Mitchell Hentges
Julien Cristau
Narcis Beleuzu
Bogdan Tara
Kimberly Sereduck
Steve Fink
Christian Holler
Butkovits Atila
Csoregi Natalia
Jeff Gilbert
Andrew Halberstadt
stransky
Sylvestre Ledru
Ricky Stewart
Olivier Tilloy
Cristina Coroiu
Andrew Creskey
Johan Lorenzo
Frederik Braun
Jean-Yves Avenard
Andrew Halberstadt
Rob Lemley