To build native python packages, the python dev packages are needed.
This should resolve the psutil installation failure.
Differential Revision: https://phabricator.services.mozilla.com/D98197
Install `psutil` when setting up the `mach` `virtualenv`s and stop importing the in-tree version in the build.
Nothing in-tree currently assumes or mandates the installation of `psutil` (all uses of `psutil` are guarded with imports of the form `try : import psutil; except ImportError: psutil = None`), so there's no back-incompatibility concerns here. There will be an awkward period where telemetry will be lacking CPU/disk data for everyone until they re-run `mach bootstrap` or `mach create-mach-environment`, but that will come back as people gradually update their `virtualenv`s.
An alternative to circumvent that issue is REQUIRING that `psutil` be installed by adding an assertion in `mach` that `psutil` can be found (allowing us to remove all the conditional logic in-tree around whether `psutil` is installed), but I wouldn't claim that we're ready to do that and deal with whatever fallout might occur.
Differential Revision: https://phabricator.services.mozilla.com/D90914
To avoid breakage from Ubuntu package upgrades, we split the test image
into two, one that sets up the packages only, and that won't change when
we need to update our scripts, and another that derives from it, that
adds all our scripts and other setup.
Additionally, we work around the recent timeout issues due to the
upgrade of packages.
The timeout itself is due to gst-launch waiting indefinitely when it
crashes, rather than exiting with an error code. Bug 1679491 addresses
this issue, but the core problem is that gst-launch crashes, which seems
to be that some change in libc (presumably "Fix pthread_rwlock_try*lock
stalls") turns `gst_object_unref: assertion '((GObject *)
object)->ref_count > 0' failed` fatal warnings (which were already
happening) into actual crashes (presumably because a race condition is
lost on a use-after-free).
This workaround, however, will stop working as soon as the updated libc
package migrates from bionic-updates into bionic proper, presumaby on
the next 18.04 dot-release. Hopefully, we won't be rebuilding the base
image for a while, avoiding further problems. Eventually, we'll want to
upload the base image to docker hub so that it's set in stone, and
change the FROM in the base image to use that instead.
Differential Revision: https://phabricator.services.mozilla.com/D98045
This adds a linux64-rust-dev toolchain and a git fetch of rust-lang/rust
that it can source from (stable tag for 1.47).
There are some issues with cross-compiling, so for now the toolchain only
builds a host build, although a lot of the machinery for cross compiling is
there for anyone brave/desperate enough to get it working.
Also note some changes were made to Rust's config.toml between 1.47 and 1.50,
so some version detection may need to be added in the future.
There is experimental support for providing patches via a new --patch flag.
Additionally, I documented the existence of the "bors-" mode.
Differential Revision: https://phabricator.services.mozilla.com/D97497
Install `psutil` when setting up the `mach` `virtualenv`s and stop importing the in-tree version in the build.
Nothing in-tree currently assumes or mandates the installation of `psutil` (all uses of `psutil` are guarded with imports of the form `try : import psutil; except ImportError: psutil = None`), so there's no back-incompatibility concerns here. There will be an awkward period where telemetry will be lacking CPU/disk data for everyone until they re-run `mach bootstrap` or `mach create-mach-environment`, but that will come back as people gradually update their `virtualenv`s.
An alternative to circumvent that issue is REQUIRING that `psutil` be installed by adding an assertion in `mach` that `psutil` can be found (allowing us to remove all the conditional logic in-tree around whether `psutil` is installed), but I wouldn't claim that we're ready to do that and deal with whatever fallout might occur.
Differential Revision: https://phabricator.services.mozilla.com/D90914
CLOSED TREE
Backed out changeset baa1f7a615e4 (bug 1666345)
Backed out changeset b6646baa866d (bug 1661624)
Backed out changeset e4d550db6037 (bug 1667152)
Install `psutil` when setting up the `mach` `virtualenv`s and stop importing the in-tree version in the build.
Nothing in-tree currently assumes or mandates the installation of `psutil` (all uses of `psutil` are guarded with imports of the form `try : import psutil; except ImportError: psutil = None`), so there's no back-incompatibility concerns here. There will be an awkward period where telemetry will be lacking CPU/disk data for everyone until they re-run `mach bootstrap` or `mach create-mach-environment`, but that will come back as people gradually update their `virtualenv`s.
An alternative to circumvent that issue is REQUIRING that `psutil` be installed by adding an assertion in `mach` that `psutil` can be found (allowing us to remove all the conditional logic in-tree around whether `psutil` is installed), but I wouldn't claim that we're ready to do that and deal with whatever fallout might occur.
Differential Revision: https://phabricator.services.mozilla.com/D90914
Install `psutil` when setting up the `mach` `virtualenv`s and stop importing the in-tree version in the build.
Nothing in-tree currently assumes or mandates the installation of `psutil` (all uses of `psutil` are guarded with imports of the form `try : import psutil; except ImportError: psutil = None`), so there's no back-incompatibility concerns here. There will be an awkward period where telemetry will be lacking CPU/disk data for everyone until they re-run `mach bootstrap` or `mach create-mach-environment`, but that will come back as people gradually update their `virtualenv`s.
An alternative to circumvent that issue is REQUIRING that `psutil` be installed by adding an assertion in `mach` that `psutil` can be found (allowing us to remove all the conditional logic in-tree around whether `psutil` is installed), but I wouldn't claim that we're ready to do that and deal with whatever fallout might occur.
Differential Revision: https://phabricator.services.mozilla.com/D90914
This patch changes a few different things:
- The Docker image used for both scrapers are now based off of our standard
Debian 10 image instead of an old Ubuntu one
- Both images were changed to make it easy to work with them when an
interactive task is used
- The python packages used by the scripts had their versions pinned and
cryptographic hashes have been recorded so that they can be verified before
running the task
- The dump_syms version used on Windows was bumped to include fixes that allow
us to scrape public symbols from DLLs and EXEs that have no PDB file available
- The maximum open file limit of the Window scraper was bumped up to cope with
runs where we gather plenty of symbols in one go
- Similarly the maximum number of symultaneous connections to a single symbol
server has been limited to 4 to avoid being throttled
- The macOS Python tools were modified to work with Python 3
Differential Revision: https://phabricator.services.mozilla.com/D95853
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Older versions of kaniko didn't handle $PATH correctly in `RUN`
commands, and we worked around this by using full paths for the
executables. Now that the base image builder is upgraded to kaniko 1,
we can remove those workarounds.
Differential Revision: https://phabricator.services.mozilla.com/D88932
`flake8`, `black`, and `pylint` are all installed in `virtualenv`s when running `mach lint`, so installing these packages globally on image setup isn't necessary.
Differential Revision: https://phabricator.services.mozilla.com/D87904
This patch fixes an issue in the custom-v8 toolchain where there is tooling missing that is needed to build v8 (the libtinfo5 package).
Differential Revision: https://phabricator.services.mozilla.com/D88310
Fixing the font cache issue the 3rd time now by just running "fc-cache" as the worker user should hopefully fix it for real, and no longer be regressed by some docker image changes.
The Puppeteer jobs will keep the 15s startup timeout for Firefox, so that we could detect another fallout again.
Differential Revision: https://phabricator.services.mozilla.com/D87745
This patch adds a new pageload testing mode called chimera mode. This mode runs the cold and warm pageload tests together within a single browser session (cold first, warm second). Some changes had to be made in the visual metrics task to be able to do this (per-test specification of extraOptions).
Differential Revision: https://phabricator.services.mozilla.com/D83356
Changes:
- create a `test-android.sh` script that is based on, but removes non-android related paths from the `test-linux.sh` script.
Differential Revision: https://phabricator.services.mozilla.com/D82252
Alex Ionescu
ffxbld
Cosmin Sabou
Tom Ritter
Mitchell Hentges
Gregory Mierzwinski
Ricky Stewart
Mike Hommey
Olivier Tilloy
Alexis Beingessner
Ben Hearsum
Noemi Erli
Mihai Alexandru Michis
smolnar
Gabriele Svelto
Bogdan Tara
Dorel Luca
Rob Lemley
Mark Banner
Michael Pobega
Razvan Maries
Henrik Skupin
Tom Prince
Simon Fraser
Johan Lorenzo
Butkovits Atila
Edwin Takahashi