I enabled -Wshadow-uncaptured-local warnings in bug 1718408 because the flag didn't report any -Wshadow-uncaptured-local warnings. Unfortunately, clang didn't report any warnings due to clang bug https://bugs.llvm.org/show_bug.cgi?id=52325: clang -Wshadow-uncaptured-local (and some other -Wshadow*) flags doesn't actually enable these warnings; they're only enabled by the meta flags -Wshadow and -Wshadow-all.
I see now that there are over 500 -Wshadow-uncaptured-local warnings, too many to realistically fix them all, so we should remove -Wshadow-uncaptured-local.
The -Wshadow-field-in-constructor-modified flag is also affected by the clang bug, but I'd like to keep setting the -Wshadow-field-in-constructor-modified flag in case the clang bug is ever fixed. There are no -Wshadow-field-in-constructor-modified warnings in mozilla-central; I fixed the last one in bug 1738400.
Differential Revision: https://phabricator.services.mozilla.com/D132290
We already use the adoptopenjdk on some platforms, this allows us to have a
more predictable Java binary on all platforms.
Differential Revision: https://phabricator.services.mozilla.com/D130878
Build and run the Mach virtualenv from a `state_dir` that is
"specific-to-topsrcdir".
As part of this, move `get_state_dir()` to `mach` so that it's usable
before `sys.path` entries are fully set up.
Differential Revision: https://phabricator.services.mozilla.com/D130383
mach will initialize from comm/build/mach_initialize.py if it's present.
See bug 1731160. The initialization function wraps the one in build/mach_initialize.py,
then extends sys.path and loads more mach_commands.py files
Differential Revision: https://phabricator.services.mozilla.com/D131869
Consolidate Mach virtualenv management to the front of the
Mach process. This obsoletes `./mach create-mach-environment`
and simplifies the `sh` portion of the top-level `./mach` script.
This helps ensure that the Mach virtualenv doesn't become
out-of-sync and simplifies the mental model of the Mach
virtualenv situation.
Differential Revision: https://phabricator.services.mozilla.com/D120401
`--no-virtualenv` was needed for one use case: the
`get_and_diffoscope` task, despite not needing `psutil`, would
run into failures during virtualenv-creation because we used to
unconditionally build the `psutil` package, and `get_and_diffoscope`
didn't have the environment needed for such a build.
Since we no longer build and install `psutil` into every virtualenv,
it's no longer needed for its one usage, which means that it can be
removed.
`--requirements` is replaced by `--virtualenv`, which removes an
ad-hoc pip package installation and embraces the centralized dep
system.
`--no-activate` is now implied by default: a virtualenv is only
created and activated if `--virtualenv` is provided.`
`ipython==7.16.1` was the chosen version because it is the last
one compatible with Python 3.6.
Differential Revision: https://phabricator.services.mozilla.com/D131529
At Mach initialization time, it needs to know if it's not running from
the Mach virtualenv so it can decide to insulate itself from the
external sys.path.
However, the current mechanism of detecting the Mach virtualenv
struggles if the venv is old, and therefore missing its metadata file.
In such a case, it's recognized as "not the Mach venv" rather than "an
out-of-date Mach venv".
As part of this, I'm realizing that simply using the metadata file as a
"virtualenv has finished building" marker is insufficient, because it
will cause similar confusion here. This is solved with a "finalization"
key in in the metadata.
Differential Revision: https://phabricator.services.mozilla.com/D130794
This is needed because `black==21.11b1` requires `click>=7.1.2`
`pathspec<1,>=0.9.0`, and `typing-extensions>=3.10.0.0`.
Differential Revision: https://phabricator.services.mozilla.com/D127163
The revert doesn't have a lot of impact, as it's only for debug info
about enums with a 128-bits integer representation.
Differential Revision: https://phabricator.services.mozilla.com/D132001
It assumes 16-bytes alignment for small allocations on 64-bits, which is
not true for mozjemalloc allocations.
This only addresses the problem with mozilla.org-produced builds. We may
have to figure out something for downstreams.
Differential Revision: https://phabricator.services.mozilla.com/D131883
This is needed because `black==21.11b1` requires `click>=7.1.2`
`pathspec<1,>=0.9.0`, and `typing-extensions>=3.10.0.0`.
Differential Revision: https://phabricator.services.mozilla.com/D127163
Short of solving this by fully embracing the monorepo build and removing
the projects/ workarounds (which will also require a compat more with
older clang versions we still build with the same script), we can solve
the bustage with a small hack.
Differential Revision: https://phabricator.services.mozilla.com/D131458
The command site manager needs to be able to do ad-hoc pip
installations, while the Mach site manager needs to manage
the system `sys.path` and conditionally create an on-disk
virtualenv.
By splitting the class into two, we can now give each use case the
attention it deserves.
Differential Revision: https://phabricator.services.mozilla.com/D129529
Sorry for the flip-flop on technique here :S
`validate_environment_packages()` was originally run when checking if a
site is up-to-date to ensure that ad-hoc pip installs didn't replace
needed packages with those of different versions.
However, since it was added, a few notes have come up:
1. The case where requirements change isn't caught by this - that is
caught earlier by the cheap "a requirements file has changed on-disk"
check.
2. This is really slow, and doing it for most Mach commands is not worth
it (as evident by how the `skip_pip_package_check` was already added
for the Mach site's use case).
3. Since the tree as-is doesn't have (common) cases where ad-hoc
installations break an environment, then this check, though helpful,
isn't adding a significant amount of value considering its performance
cost.
However, these aren't to say that this won't be valuable in the future:
I'd like to reach a point where sites are considered "sealed" by
default: no ad-hoc pip installations are allowed.
However, add the ability to mark sites as unsealed/"allowing
ad-hoc pip installations". Then, re-add the pip package check, but only
for such flexible, unsealed virtualenvs.
Differential Revision: https://phabricator.services.mozilla.com/D129692
The current inheritance-based separation between "general venv
details" (`VirtualenvHelper`) and moz site details
(`MozSiteManager`) has caused `MozSiteManager` to care
about too many details.
This patch splits things apart a little more:
* The term `VirtualenvHelper` isn't _super_ useful, rename it to
`PythonVenv` to show that it represents a Python virtualenv.
* Move on-disk virtualenv logic (`activate_path` location,
`site-packages` location, `pip install` behaviour) into a
contained `MozVirtualenv` class.
* Port the inheritance to a "composition" model instead.
Differential Revision: https://phabricator.services.mozilla.com/D129685
The existing terminology had two issues:
* `VirtualenvManager` wasn't always associated with an on-disk
`virtualenv`: for example, when running in automation, Mach
"activates" a `VirtualenvManager`, updating its import scope,
but without ever creating an on-disk `virtualenv`.
* An upcoming patch splits the `VirtualenvManager` class, pulling
"on-disk virtualenv-handling functions" from the project-wide
interface for managing Python's import scope.
After some good discussion with Ahal, I think we've struck
the terminology that handles this distinction well: we'll call
the "import scope"-handling part the "site", and we'll continue
to call on-disk virtualenvs (and their representative classes)
as, well, virtualenvs.
Differential Revision: https://phabricator.services.mozilla.com/D130391
Otherwise, the warning is displayed:
```
WARNING: Could not lex literal_block as "json". Highlighting skipped.
```
Depends on D131092
Differential Revision: https://phabricator.services.mozilla.com/D131093
It's possible for the `PYTHON3` config to point to a different Python
than that that is executing the configure scripts themselves.
This flexibility was needed for the Python 2->3 migration, but now that
it's complete, we can remove the extra configuration and just lean on
the Python interpreter used to run configure.
Differential Revision: https://phabricator.services.mozilla.com/D129863
There were a bunch of locations where we were doing path shenanigans
with `requirements.pth/.vendored` items.
There was a bit of complexity because we were specifically making each
`pthfile` line be a relative path to support moving the Firefox
topsrcdir without causing issues.
However, now that we're more intelligent about checking if `pthfile`
lines are up-to-date (and since moving your topsrcdir will still require
re-generating the Mach virtualenv), this behaviour became less useful.
So, generalize `MachEnvRequirements` -> "sys.path lines" logic and
reuse it everywhere.
Differential Revision: https://phabricator.services.mozilla.com/D129693
`tqdm` allows rendering and updating a single-line progress bar, which
is useful for all sorts of different work.
Our first use case for this will be replacing `wget` with a pure-python
downloader.
Source here: https://github.com/tqdm/tqdm
Differential Revision: https://phabricator.services.mozilla.com/D129094
The optional `log_handle` argument was only used by:
* Configure, but the output was always dumped to stdout and *not*
`config.log`. The manual logging _was_ used to handle encoding issues,
but those are likely invalidated by the Python 3 migration.
* `./mach doc`, where we were putting virtualenv setup into stderr,
which seems incorrect. The commit adding it doesn't explain why it's
the case, but I'm guessing it shouldn't be too risky to remove.
Additionally, `log_handle` was used very inconsistently: for example,
running `install_pip_package()` would _not_ use `log_handle`.
So, removing `log_handle` removes a bit of abstraction leakage.
Differential Revision: https://phabricator.services.mozilla.com/D129298
Mach already verifies the Python version during initialization.
Maintaining a second version check can be tough to keep up-to-date, as
we're already seeing due to the obsolete Python 2 check.
Differential Revision: https://phabricator.services.mozilla.com/D129297
Now that are prioritizing system over virtualenv site-packages, the
system `pip` is sometimes being used instead.
This is causing issues when the system pip is set up in a
distro-specific way, such as when "debundled":
https://github.com/pypa/pip/blob/9.0.1/pip/_vendor/__init__.py#L53-L61
However, if we vendor `pip`, `setuptools` and `wheel`, and ensure that
they're prioritized in the `sys.path` before anything is imported from
the system, then we can ensure that we're using a modern `pip` _and_
sidestep system-specific pip weirdness.
Note that `pip-compile`'s `--allow-unsafe` flag is not as dangerous as
it sounds.
There's confusion among maintainers about its origin:
https://github.com/jazzband/pip-tools/issues/522
Additionally, it's going to be enabled by default in a future
`pip-tools` release. So, it's not scary for us to embrace here.
Also, heads up that the "pip outdated warning" no longer needs
to be manually silenced, since pip avoids that code path when
not running from an "installed" context.
Differential Revision: https://phabricator.services.mozilla.com/D127182
The previous behaviour was to:
* Never add a `pthfile` to the Mach virtualenv, and
* Always add Mach's paths to the `sys.path` when Mach initializes
However, this meant that `pip` would needlessly install packages
that already exist in the vendored environment.
Tweak `pth` behaviour so that `pip` behaves more efficiently.
Differential Revision: https://phabricator.services.mozilla.com/D120402
As we leverage the Mach environment more, it becomes increasingly
important that it isn't out-of-date on developer machines.
Add an `up_to_date()` check during Mach initialization.
To minimize the cost to startup, I'm skipping the "pip list" check.
This change required moving `virtualenv` from `mozbuild` to `mach` to
make it available during the early stage of Mach init.
Differential Revision: https://phabricator.services.mozilla.com/D127144
Chris Peterson
Mike Hommey
Agi Sferro
Mitchell Hentges
Rob Lemley
Christian Holler
Shravan Narayan
Butkovits Atila
Marian-Vasile Laza
Alexandre Lissy
Alexandru Michis
Benjamin Beurdouche
Sylvestre Ledru
Kershaw Chang
Niklas Goegge
Alex Hochheiden
Sebastian Hengst