Bug 1683824 updated to the 0.2.14 tag, but that's patched in order to
publish to crates.io such that tiny-http is not patched anymore.
Unfortunately, that breaks sccache-dist, which is a known issue and will
eventually fixed when addressing
https://github.com/mozilla/sccache/issues/912.
So here we update to a corresponding version on master (the first that
bumps the version to the next number).
Differential Revision: https://phabricator.services.mozilla.com/D100863
I was waiting for a better reason to do this, because the cbindgen
changes from 0.15.0 to 0.16.0 don't break trunk builds. But since
downstream has updated (see bug 1684180) and there's no reason not to,
let's do this to avoid future churn.
Differential Revision: https://phabricator.services.mozilla.com/D100499
Since e10s-multi is going to be enabled by default, this gives us the ability
to run junit in a secondary, e10s-single configuration so that we can see them
both side-by-side for a little while, at least until we're settled in with the
e10s-multi configuration.
Differential Revision: https://phabricator.services.mozilla.com/D94883
When building compiler-rt for macOS, its build system assumes the
compiler used is a native macOS clang, drops all the cmake C/C++ flags,
and adds its own.
Because we do need to pass `-target $target` (otherwise we end up with
ELF x86_64 objects) and `-mcpu=apple-a12` for the correct baseline for
arm64 macOS, and because there is unfortunately no cmake variable that
the compiler-rt build system will use, and because CMAKE_C*_COMPILER
need to be a program without arguments, we need to wrap the compiler.
While here, add `-v` to the ninja call to have more useful logs.
Differential Revision: https://phabricator.services.mozilla.com/D99841
This patch enables ASan for rust code in the `linux64-asan/{debug,opt}` and `linux64-asan-fuzzing` build tasks. Since it is a `-Z` unstable flag, we'll move to the same `linux64-rust-nightly` toolchain that TSan builds use.
For the time being, the ASan Nightly Reporter builds are left alone, until we're more familiar with the perf impacts of this landing.
Differential Revision: https://phabricator.services.mozilla.com/D99556
At this point it's pretty clear that we won't be reverting to clang-9.
This doesn't remove everything with clang-9 in the name. I did a mark-and-sweep GC by hand so this only removes unused entries. Some are still active, e.g. linux64-clang-9 is used to build a number of misc helper tools.
Differential Revision: https://phabricator.services.mozilla.com/D99721
This requires tests to specify `python-3: true` in order to be run
with Python 3. When nothing is specified things work just like today,
so it's a more conservative change than the one in bug 1672181.
Obviously in the long term we will remove this and switch to Python 3
only, but this unblocks moving harnesses to py3 today.
Differential Revision: https://phabricator.services.mozilla.com/D98766
Instantiating a wasm library duplicates a file descriptor for /dev/null 3 times to be used as input, output and error streams for the wasm sandboxed code. When a lot of sandboxes are created and destroyed, a lot of descriptors are duplicated and closed. While this should be fine, POSIX does not seem to happy with the opening and closing of many file descriptors --- this could perhaps be some strange interaction with Firefox's seccomp filters and cross-process file descriptor handling as it is difficult to repro this outside of firefox.
However, the simpler fix here was to just eliminate the duplication of /dev/null and return an error when input, output or error streams are accessed by wasm sandboxed code. This means calls to printf will fail, but no code I know off actually checks the int error code returned by printf and this change is certainly compatible with existing sandboxed components.
Differential Revision: https://phabricator.services.mozilla.com/D99160
This requires tests to specify `python-3: true` in order to be run
with Python 3. When nothing is specified things work just like today,
so it's a more conservative change than the one in bug 1672181.
Obviously in the long term we will remove this and switch to Python 3
only, but this unblocks moving harnesses to py3 today.
Differential Revision: https://phabricator.services.mozilla.com/D98766
This is step 1 to make the docs here better, as discussed. Another followup
will come in a day or two to update the text/links of the docs to refer to
this and the latest version of repack-rust.
Differential Revision: https://phabricator.services.mozilla.com/D99128
We ended up with a symbol `mda2-dafa26b89ed-bk-1754415059f-bk`, which is longer
than the allowed schema length of 25 characters. Let's keep the existing
backfill suffix per sheriffs.
Differential Revision: https://phabricator.services.mozilla.com/D98694
Triggered by the platform name change in bug 1673071 when macOS switched to
running its tests with WebRender enabled.
Differential Revision: https://phabricator.services.mozilla.com/D99095
clang 12 (specifically https://reviews.llvm.org/D91379) made some refactorings to libc++ that exposed a problem in the MinGW headers. That has now been fixed upstream.
In the meantime the headers also gained definitions for ProcessPayloadRestrictionPolicy, so we can remove our workaround for that.
Differential Revision: https://phabricator.services.mozilla.com/D98945
To build native python packages, the python dev packages are needed.
This should resolve the psutil installation failure.
Differential Revision: https://phabricator.services.mozilla.com/D98197
Removes a bunch of zstandard installations that were ran from
within the mach virtualenv, which should already have zstandard
installed.
Differential Revision: https://phabricator.services.mozilla.com/D98387
To ensure l10n updates are still picked up by beta builds in a timely
fashion, remove "DONTBUILD" from commit messages when running on
mozilla-beta, and run a couple of hours before the "daily-releases" job
starts.
Differential Revision: https://phabricator.services.mozilla.com/D98349
We keep dump-syms on 1.47 because it breaks at runtime when built with
1.48. It also puts it in sync with other toolchains that won't upgrade
until we need them to.
Differential Revision: https://phabricator.services.mozilla.com/D98422
The zstandard package is always installed in the mach virtualenv.
The patch assumes that zstandard is only used from the mach virtualenv,
and never the build virtualenv.
Differential Revision: https://phabricator.services.mozilla.com/D98387
While all toolkit and js-based projects make use of mfbt, some others,
like tools/crashreporter and tools/update-packaging, don't.
So instead of including mfbt from the top-level directory, include it
from the relevant project top-level mozbuilds.
This allows to remove the dependency on mfbt files in the hash for the
minidump-stackwalk and mar-tools toolchains.
Differential Revision: https://phabricator.services.mozilla.com/D98378
To ensure l10n updates are still picked up by beta builds in a timely
fashion, remove "DONTBUILD" from commit messages when running on
mozilla-beta, and run a couple of hours before the "daily-releases" job
starts.
Differential Revision: https://phabricator.services.mozilla.com/D98349
Bug 1672888 set up a plain opt build that runs on m-c only, like the
other plain opt builds. It didn't add plain debug builds on integration
branches as well to avoid having too many jobs running on the slim
mac worker pool, but since bug 1678154 moved most jobs off them, we can
now add those.
Differential Revision: https://phabricator.services.mozilla.com/D98279
Install `psutil` when setting up the `mach` `virtualenv`s and stop importing the in-tree version in the build.
Nothing in-tree currently assumes or mandates the installation of `psutil` (all uses of `psutil` are guarded with imports of the form `try : import psutil; except ImportError: psutil = None`), so there's no back-incompatibility concerns here. There will be an awkward period where telemetry will be lacking CPU/disk data for everyone until they re-run `mach bootstrap` or `mach create-mach-environment`, but that will come back as people gradually update their `virtualenv`s.
An alternative to circumvent that issue is REQUIRING that `psutil` be installed by adding an assertion in `mach` that `psutil` can be found (allowing us to remove all the conditional logic in-tree around whether `psutil` is installed), but I wouldn't claim that we're ready to do that and deal with whatever fallout might occur.
Differential Revision: https://phabricator.services.mozilla.com/D90914
To avoid breakage from Ubuntu package upgrades, we split the test image
into two, one that sets up the packages only, and that won't change when
we need to update our scripts, and another that derives from it, that
adds all our scripts and other setup.
Additionally, we work around the recent timeout issues due to the
upgrade of packages.
The timeout itself is due to gst-launch waiting indefinitely when it
crashes, rather than exiting with an error code. Bug 1679491 addresses
this issue, but the core problem is that gst-launch crashes, which seems
to be that some change in libc (presumably "Fix pthread_rwlock_try*lock
stalls") turns `gst_object_unref: assertion '((GObject *)
object)->ref_count > 0' failed` fatal warnings (which were already
happening) into actual crashes (presumably because a race condition is
lost on a use-after-free).
This workaround, however, will stop working as soon as the updated libc
package migrates from bionic-updates into bionic proper, presumaby on
the next 18.04 dot-release. Hopefully, we won't be rebuilding the base
image for a while, avoiding further problems. Eventually, we'll want to
upload the base image to docker hub so that it's set in stone, and
change the FROM in the base image to use that instead.
Differential Revision: https://phabricator.services.mozilla.com/D98045
This adds a linux64-rust-dev toolchain and a git fetch of rust-lang/rust
that it can source from (stable tag for 1.47).
There are some issues with cross-compiling, so for now the toolchain only
builds a host build, although a lot of the machinery for cross compiling is
there for anyone brave/desperate enough to get it working.
Also note some changes were made to Rust's config.toml between 1.47 and 1.50,
so some version detection may need to be added in the future.
There is experimental support for providing patches via a new --patch flag.
Additionally, I documented the existence of the "bors-" mode.
Differential Revision: https://phabricator.services.mozilla.com/D97497
And don't set SHELL on mac workers (added in da452e43b5d5 because of the
exception thrown by this code not having a fallback)
Differential Revision: https://phabricator.services.mozilla.com/D97686
Backfill tasks might run with a slightly different definition than the task
with the same label in the current task graph. When we encounter them, ensure
we create a new task with an (almost) identical definition, rather than
grabbing it from the current push's full-task-graph.json.
There's already an action that has this behaviour (that is used for the
decision task). So rather than re-implement this behaviour in the regular
retrigger action, this patch ensures that the 'retrigger-decision' action
applies to backfills rather than the regular one.
Differential Revision: https://phabricator.services.mozilla.com/D97352
According to the documentation, actions are supposed to have an "order" system,
where the first action that applies to a task via its context, is supposed to
take precedence. But when running the match algorithm, we never break out of
the loop. So we end up applying the *last* action that matches a tasks' context.
As far as I can tell, this means we have been using the 'retrigger-disabled'
action for everything and things like retriggering Decision tasks don't work.
Differential Revision: https://phabricator.services.mozilla.com/D97351
If we aren't able to assign the appropriate manifests to the backfill, the
backfill is useless. Let's just let the exception propagate and fail loudly,
rather than log an error and keep going.
Differential Revision: https://phabricator.services.mozilla.com/D97350
remove default scheduling of talos, raptor, awsy on windows7-32. Ensure these are still available on try server. There is one exception, the talos-webgl-gli job, this is scheduled via a transform and I would rather leave that alone or remove all gli tests.
Differential Revision: https://phabricator.services.mozilla.com/D97310
By using a string instead of list for the command, we leave it to
the run-task transform to wrap with bash, which we don't need to do
ourselves anymore.
Differential Revision: https://phabricator.services.mozilla.com/D96963
Currently, if a task defines its own expiry with a very large value,
that will be respected even on try, where we actually don't want that to
happen.
This also helps simplify the setup for docker images.
We also take on the occasion to remove the discrepancy between the
default expiry for tasks in general and tests in particular. Bug 1258497
set the original expiry to 14 days, bug 1281004 added another place
where the expiry was set to 14 days for tests specifically, and then bug
1304180 changed the expiry to 28 days, but it just seems the location
for tests was overlooked rather than deliberately left to 14 days.
Differential Revision: https://phabricator.services.mozilla.com/D96962
Also fix the gecko path on extra-config-path flags on mac tasks
(currently not causing problems because unused).
And while here, we switch to python3.
Differential Revision: https://phabricator.services.mozilla.com/D96961
* Bumps the tsan toolchain to rust-nightly-2020-11-14 that has my patches to make -Zbuild-std work in vendored environments:
* https://github.com/rust-lang/cargo/pull/8834
* https://github.com/rust-lang/rust/pull/78790
* Passes -Zbuild-std to cargo when MOZ_TSAN is defined (mk_add_options --enable-thread-sanitizer)
* Removes generic Rust supressions and adds much more specific ones
* One presumed upstream false positive from tsan not understanding the code
* One actual upstream bug tsan found (yay!)
* One new real issue uncovered
* One issue that probably already existed intermittently but I happened to hit
Differential Revision: https://phabricator.services.mozilla.com/D97165
Install `psutil` when setting up the `mach` `virtualenv`s and stop importing the in-tree version in the build.
Nothing in-tree currently assumes or mandates the installation of `psutil` (all uses of `psutil` are guarded with imports of the form `try : import psutil; except ImportError: psutil = None`), so there's no back-incompatibility concerns here. There will be an awkward period where telemetry will be lacking CPU/disk data for everyone until they re-run `mach bootstrap` or `mach create-mach-environment`, but that will come back as people gradually update their `virtualenv`s.
An alternative to circumvent that issue is REQUIRING that `psutil` be installed by adding an assertion in `mach` that `psutil` can be found (allowing us to remove all the conditional logic in-tree around whether `psutil` is installed), but I wouldn't claim that we're ready to do that and deal with whatever fallout might occur.
Differential Revision: https://phabricator.services.mozilla.com/D90914
WebGPU itself doesn't care about WebRender or even the GPU process.
However, the presentation to canvas as written today only works with WebRender, so
it seems fine to have it as a dependency in general.
An alternative could be to just fail getContext() call if webrender is disabled.
Differential Revision: https://phabricator.services.mozilla.com/D70888
A bit counterintuitive because we're keeping Warp and not Ion, but it's easiest
to make this change and later potentially rename the Ion prefs to Warp prefs.
Depends on D96987
Differential Revision: https://phabricator.services.mozilla.com/D96988
CLOSED TREE
Backed out changeset baa1f7a615e4 (bug 1666345)
Backed out changeset b6646baa866d (bug 1661624)
Backed out changeset e4d550db6037 (bug 1667152)
Install `psutil` when setting up the `mach` `virtualenv`s and stop importing the in-tree version in the build.
Nothing in-tree currently assumes or mandates the installation of `psutil` (all uses of `psutil` are guarded with imports of the form `try : import psutil; except ImportError: psutil = None`), so there's no back-incompatibility concerns here. There will be an awkward period where telemetry will be lacking CPU/disk data for everyone until they re-run `mach bootstrap` or `mach create-mach-environment`, but that will come back as people gradually update their `virtualenv`s.
An alternative to circumvent that issue is REQUIRING that `psutil` be installed by adding an assertion in `mach` that `psutil` can be found (allowing us to remove all the conditional logic in-tree around whether `psutil` is installed), but I wouldn't claim that we're ready to do that and deal with whatever fallout might occur.
Differential Revision: https://phabricator.services.mozilla.com/D90914
This makes --enable-thread-sanitizer turn on Rust tsan (-Zsanitizer=thread).
This requires changing SpiderMonkey tsan to use the tsan rust nightly.
In future changes, more Rust tsan integration will key off of MOZ_TSAN.
Differential Revision: https://phabricator.services.mozilla.com/D96453
Install `psutil` when setting up the `mach` `virtualenv`s and stop importing the in-tree version in the build.
Nothing in-tree currently assumes or mandates the installation of `psutil` (all uses of `psutil` are guarded with imports of the form `try : import psutil; except ImportError: psutil = None`), so there's no back-incompatibility concerns here. There will be an awkward period where telemetry will be lacking CPU/disk data for everyone until they re-run `mach bootstrap` or `mach create-mach-environment`, but that will come back as people gradually update their `virtualenv`s.
An alternative to circumvent that issue is REQUIRING that `psutil` be installed by adding an assertion in `mach` that `psutil` can be found (allowing us to remove all the conditional logic in-tree around whether `psutil` is installed), but I wouldn't claim that we're ready to do that and deal with whatever fallout might occur.
Differential Revision: https://phabricator.services.mozilla.com/D90914
The pref name has changed from wasm_ionjit to wasm_optimizingjit
(partly to accomodate macOS, partly because this is better for the
current architecture). So rename the prefs, and also rename the tests
to reflect this reality. A PR has been submitted to update awfy.
Differential Revision: https://phabricator.services.mozilla.com/D96376
If we don't find any revisions in !public() - when run on autoland - then the
assumptions of the script are incorrect and we may have changed something, such
that the script starts silently succeeding even though it's not checking anything.
Differential Revision: https://phabricator.services.mozilla.com/D95591
This makes --enable-thread-sanitizer turn on Rust tsan (-Zsanitizer=thread).
This requires changing SpiderMonkey tsan to use the tsan rust nightly.
In future changes, more Rust tsan integration will key off of MOZ_TSAN.
Differential Revision: https://phabricator.services.mozilla.com/D96453
Since e10s-multi is going to be enabled by default, this gives us the ability
to run junit in a secondary, e10s-single configuration so that we can see them
both side-by-side for a little while, at least until we're settled in with the
e10s-multi configuration.
Differential Revision: https://phabricator.services.mozilla.com/D94883
When the virtualenv is activated, subprocesses of mozharness inherit the
PATH that it sets, making plain python resolve to the virtualenv python,
which is actually not desirable for builds.
Ideally, we'd avoid creating a virtualenv for mozharness entirely,
especially considering it creates its own, and then the build creates
yet another one, but that doesn't work because of PYTHONPATH interfering
with the build virtualenv.
Differential Revision: https://phabricator.services.mozilla.com/D96168
This patch changes a few different things:
- The Docker image used for both scrapers are now based off of our standard
Debian 10 image instead of an old Ubuntu one
- Both images were changed to make it easy to work with them when an
interactive task is used
- The python packages used by the scripts had their versions pinned and
cryptographic hashes have been recorded so that they can be verified before
running the task
- The dump_syms version used on Windows was bumped to include fixes that allow
us to scrape public symbols from DLLs and EXEs that have no PDB file available
- The maximum open file limit of the Window scraper was bumped up to cope with
runs where we gather plenty of symbols in one go
- Similarly the maximum number of symultaneous connections to a single symbol
server has been limited to 4 to avoid being throttled
- The macOS Python tools were modified to work with Python 3
Differential Revision: https://phabricator.services.mozilla.com/D95853
This patch will make sure that we select the right
wheels for numpy and scipy when `--visualmetrics` is used,
so we don't need to compile them.
It also make sure we don't install all those packages
unless the environment wants to use visual metrics.
Differential Revision: https://phabricator.services.mozilla.com/D91740
This patch makes webrender-enabled testing the main method for testing and also adds a subset of tests that will without it enabled until the switch is fully complete.
Differential Revision: https://phabricator.services.mozilla.com/D95744
This patch upgrade browsertime to v10, enables webdriver-based navigation in browsertime tests, and also disables the Firefox WindowRecorder for a fairer comparison between Chrome and Firefox.
Differential Revision: https://phabricator.services.mozilla.com/D95425
This patch flips things around so that the non-QR macosx test platforms are
running a "skeleton crew" set of tests, and the QR macosx test platforms are
running all the tests that were previously running with non-QR. As of Fx83
macOS has WebRender enabled by default for 90%+ of the population so it makes
sense to test that configuration more heavily, but leave some tests running
for the non-QR configuration as well.
Differential Revision: https://phabricator.services.mozilla.com/D95430
This patch flips things around so that the non-QR macosx test platforms are
running a "skeleton crew" set of tests, and the QR macosx test platforms are
running all the tests that were previously running with non-QR. As of Fx83
macOS has WebRender enabled by default for 90%+ of the population so it makes
sense to test that configuration more heavily, but leave some tests running
for the non-QR configuration as well.
Differential Revision: https://phabricator.services.mozilla.com/D95430
The migration of the task images from Ubuntu 16.04 to Ubuntu 18.04 in
bug 1602863 added the Ubuntu version to the platform. The platform of Linux x64
DevEdition didn't get added and used the default tier 2. We don't run tests
for Linux 32-bit anymore.
Differential Revision: https://phabricator.services.mozilla.com/D95754
This decreases the scheduling algorithm from a CT_MEDIUM (0.8) threshold to
CT_LOW (0.7). It also reduces the threshold used in the manifest resolving
algorithm down to 0.5. This means we'll have more "ride-along" manifests that
happen to be in the same chunk as the more important ones.
Differential Revision: https://phabricator.services.mozilla.com/D94757
The first versions with 1.48 had a regression that was fixed later
during the 1.48.0 development phase. Use the last of those to have
something closer to a release, but still versioned 1.48.
Differential Revision: https://phabricator.services.mozilla.com/D94573
Rather than putting a top-level "mozharness" directory in "mozharness.zip",
this puts the root of mozharness at the root of the archive. Then we extract
the archive into a "mozharness" directory. This way the path on disk ends up
being the same, but generic-worker won't attempt to modify the permissions of
files in the cwd.
Differential Revision: https://phabricator.services.mozilla.com/D95138
Mike Hommey
Andi-Bogdan Postelnicu
Gabriele Svelto
Jeff Muizelaar
ffxbld
Cosmin Sabou
Emilio Cobos Álvarez
Aaron Klotz
Andrew Erickson
Tarek Ziadé
David Major
Tom Ritter
Alexis Beingessner
James Graham
Jamie Nicol
Sebastian Hengst
shravanrn@gmail.com
Mihai Alexandru Michis
Aki Sasaki
Mitchell Hentges
Julien Cristau
Narcis Beleuzu
Bogdan Tara
Gregory Mierzwinski
Kimberly Sereduck
Steve Fink
Christian Holler
Butkovits Atila
Csoregi Natalia
Razvan Maries
Jeff Gilbert
Andrew Halberstadt
stransky
Sylvestre Ledru
Ricky Stewart
Olivier Tilloy
Cristina Coroiu
Joel Maher
Andrew Creskey
Johan Lorenzo
Frederik Braun
Jean-Yves Avenard
Andrew Halberstadt
Rob Lemley
Noemi Erli
Ben Hearsum
Bob Clary
Dzmitry Malyshau
Jan de Mooij
Dorel Luca
Andrew Osmond
smolnar
Brindusan Cristian
Lars T Hansen
Henrik Skupin
Chris H-C
Dave House
Julian Descottes
Kartikaya Gupta
Matt Woodrow
Agi Sferro