This was added ing bug 1352595. It has only been used briefly, and we
since then have gone all the way to never compress omni.ja. Even if we
did go back to compressing, we'd probably go with zstd rather than
brotli.
The gecko-side support for this has only ever been nightly-only.
Differential Revision: https://phabricator.services.mozilla.com/D118655
`_get_infer_config()` returns 3 items, but the existing code expects 2.
Add a third `_` to capture the other unwanted item.
Differential Revision: https://phabricator.services.mozilla.com/D118535
In the existing behaviour, we would return a node/npm executable from
`find_executable()`, even if we couldn't extract a version from it or if
the extracted version was insufficient.
Now, we fail early if the detected node/npm is insufficient.
Differential Revision: https://phabricator.services.mozilla.com/D118377
As :julienw beautifully described on Bugzilla,
we no longer need to search for "nodejs" since all
packages providing Node >= v12 will provide the `node`
binary.
See his comment here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1716600#c4
Base-toolchain tasks with Node v10 should still be
happy because, unlike Debian's Node v10, they provide
a `node` binary instead of `nodejs`.
Differential Revision: https://phabricator.services.mozilla.com/D118375
Having separate `<name>.pth` files in the virtual environments
isn't providing an advantage. We can simplify configuration
by putting all `pth` adjustments into a single file: `mach.pth`.
Differential Revision: https://phabricator.services.mozilla.com/D117710
This has two benefits:
1. `handle_package()` becomes more clear - rather than referring to
`action` and `package` with array index numbers, we now give
them real names. The benefit here is also shown in `up_to_date()`.
2. This makes the top-level parser for `packages()` less opinionated
about sub-formats: if an action has a nested structure, it should
have the flexibility to define what it looks like.
Differential Revision: https://phabricator.services.mozilla.com/D117708
After some testing in `try` and locally, the manual
`PYTHONEXECUTABLE` definitions shouldn't be needed
anymore.
There's been some work on Brew's
Python to improve its `sitecustomize` behaviour.
The most likely improvement is:
https://github.com/Homebrew/homebrew-core/pull/65297
However, I'm not fully confident in this change.
If it fails, it's more likely to affect developers
than CI. I think it's worth attempting a landing,
because if the variable is indeed obsolete, then
deleting it will avoid some spicy "action-at-a-
distance" behaviour.
Differential Revision: https://phabricator.services.mozilla.com/D117452
Vendoring wheels has three benefits:
* There's far less files, so Firefox checkouts will be smaller.
* It works around `zipp` not allowing `pip install`
from extracted source `tar.gz` files. Now, we should
be able to use the pip resolver against vendored
packages, which will be needed for future
mach virtualenv work.
* `./mach vendor python` takes far less time to execute.
Since we need the raw Python to be available to add to the `sys.path`,
we extract the wheels before putting them in tree.
Due to the structure of some wheels being less nested
than of a source `tar.gz`, `common_virtualenv_packages`
needed to be adjusted accordingly.
`install_pip_package()` had to be tweaked as well since you can't
`pip install` an extracted wheel. So, we "re-bundle" the wheel
before installing from a vendored package.
Replace python packages with wheels where possible
This contains the vendoring changes caused by the
last patch.
For reviewing, there's a couple things to note:
* A bunch of files are deleted, since there's generally
less files in a wheel than in a source archive.
* There's a new `.dist-info` directory for each
extracted wheel, so expect roughly 5 or
6 new files for each wheel'd package.
* There should be no source code changes other than
moves from package names changing from having
`-` to having `_`.
Differential Revision: https://phabricator.services.mozilla.com/D116512
Last year, we stopped vendoring Python packages that have native
code. Since we have only had pure-python packages since, the
Windows-specific qualifier (or excluder in the case of `!windows`)
hasn't been needed.
I don't foresee us needing it again, but if anything we can peel it
back from `hg` history if this assumption is incorrect.
Differential Revision: https://phabricator.services.mozilla.com/D117468
On Windows, Python's temporary directories are placed in `%APPDATA%`.
Unfortunately, the failing tests want to assume that some "other"
path (as a subpath of a tempdir) won't be within any
`filter_args()` known top-level paths, which includes the user's home
dir.
The only reason that this assumption has passed in CI so far is due
to Windows case insensitivity: running locally had `~` evaluated
to `c:\Users\...`, while running in CI had it be `c:\users\...`.
Since the tempdir path started with `C:\Users`, `filter_args()`
would `<path omitted>` in CI but would `$HOME/...` locally.
By not basing the `other_path` on `tempdir`, we now consistently get the
same results across platforms.
Differential Revision: https://phabricator.services.mozilla.com/D117698
`./mach vendor python <package>` was already adding its new package
to `requirements.txt`, so we were getting full resolver support, which
is good. However, it caused our `requirements.in` file to start
getting out-of-date, and therefore made it harder to identify the
top-level dependencies.
Arguably, we could have `./mach vendor python <package>` automatically
update `requirements.in`, too, but then we need to solve the edge cases,
such as "What if the package is already in `requirements.in`? What if
that existing item has a different version?"
The hardest part of updating `requirements.in` was finding it, so I've
also modified the `./mach vendor python` help text to make it more
identifiable.
Differential Revision: https://phabricator.services.mozilla.com/D116386
For "c" (i.e., gcc, clang, and clang-cl), this configures the
`-f{debug,macro}-path-prefix` flags. We'd prefer to use
`-ffile-path-prefix`, but it seems that `clang-cl` does not recognize
that flag.
For "rust" (i.e., rustc/cargo), this configures `--remap-path-prefix`.
This is one step toward getting `sccache` hits across source and
object directories.
Differential Revision: https://phabricator.services.mozilla.com/D113065
Based on the docs and the code within `pathlib2`, it seems to focus
only on backporting the main features of `pathlib` to be available for
Python 2. It does _not_ provide features in newer Python versions
of `pathlib` (such as `is_mount`, new in Python 3.7). Therefore, it
doesn't provide anything that the standard library of Python 3.6+
doesn't have.
Differential Revision: https://phabricator.services.mozilla.com/D117071
Adds a `mach doctor` check to assert Python's locale is not set to
`UTF-8`, an invalid value due to a Python bug in most pre-3.8 versions.
Differential Revision: https://phabricator.services.mozilla.com/D117081
Adds a check that SSH access to hg.mozilla.org is properly configured.
Attempt to run `ssh hg.mozilla.org` which will authorize to hgmo over
SSH via `pash`, without running any commands on the service. In the case
the user has properly configured SSH we will see a success message and
information on the SCM levels granted to the user. If we don't see
a successful connection, parse the output to ensure the user is using
an email address as their username (which is required on hg.mozilla.org).
If the user couldn't connect yet does have an email address as their
username, notify them that they do not have permission to push to hgmo.
Differential Revision: https://phabricator.services.mozilla.com/D117080
Now that we configure pip within mach virtualenvs to turn off
its "outdated" warning, we no longer need to apply the suppression
as a CLI argument.
Differential Revision: https://phabricator.services.mozilla.com/D116891
We don't offer API splits any more, and with the separation of GeckoView with
the rest of the front-end it's increasingly unlikely that we will in the
future.
This change makes it so that the build name doesn't contain the API version so
that we can update it without breaking all the automation that relies on the
build name.
Differential Revision: https://phabricator.services.mozilla.com/D114369
We don't offer API splits any more, and with the separation of GeckoView with
the rest of the front-end it's increasingly unlikely that we will in the
future.
This change makes it so that the build name doesn't contain the API version so
that we can update it without breaking all the automation that relies on the
build name.
Differential Revision: https://phabricator.services.mozilla.com/D114369
Adds test to ensure that we don't modify files that are vendored
with PyPI.
If a modification needs to happen to a vendored library, we should
instead:
* Modify the library upstream and vendor the new version, or
* Remove the library from requirements.in and requirements.txt so that
it's now a "manually-vendored" library. Note that there many be
issues if this occurs to a library depended on by 3rd-party
libraries.
Differential Revision: https://phabricator.services.mozilla.com/D116428
Generating Python lockfiles is system-dependent, and this currently
mostly affects us just on the Python 3.6 <=> 3.7 boundary - 3.6 requires
a few additional packages like `importlib_metadata` and `iso8601`.
If a different Python version is used, the test in CI is guaranteed to
fail. By validating in advance, we help future vendoring developers
by helping them avoid the wait before CI failure and the time it
takes to troubleshoot.
Differential Revision: https://phabricator.services.mozilla.com/D116773
Rewrite `doctor.py` from an object-oriented paradigm to a simpler
function-based architecture. Each check is a function that returns
or yields `DoctorCheck`s. `DoctorCheck`s contain a status that describes
the status of the check (success, warning, fatal error etc), a string
of text with useful information for display to the terminal, and may
optionally attach a callable which can be executed to fix issues found
by the checks.
`mach doctor` is updated to use the new `run_doctor` function instead
of creating a `Doctor` object. The default value of `verbose` in `run_doctor`
is `False`, meaning only issues with the users system are printed to the
screen.
Most of the existing check logic is left intact from the previous
implementation. Checks for `platform` are changed to use the
`sys.platform.startswith` pattern. The latest MozillaBuild version is
pulled by querying the `tip` commit on hg.mozilla.org instead of using
a hard-coded variable.
Differential Revision: https://phabricator.services.mozilla.com/D116539
HEAD is a symbolic ref that points to the default branch of the
repository, it will point to master, main, or anything else that the
repository is using as its default branch.
Differential Revision: https://phabricator.services.mozilla.com/D116678
It's not practically make a difference (the flags we end up setting it
to don't do anything when linking all the wasm files into one).
Differential Revision: https://phabricator.services.mozilla.com/D116437
Now, when running mach commands that invoke `pip`, it will no longer
inform the user that it needs an update.
We reach into `distutils` to determine the "site-packages" directory
pattern, then apply it to our virtualenv.
Differential Revision: https://phabricator.services.mozilla.com/D115940
Mike Hommey
Mitchell Hentges
Marian-Vasile Laza
surajeet310
Alexandru Michis
Csoregi Natalia
Nick Alexander
Paul Adenot
Emilio Cobos Álvarez
Andrey Bienkowski
Connor Sheehan
Steve Fink
Agi Sferro
Sandor Molnar
Dorel Luca
june wilde