The command site manager needs to be able to do ad-hoc pip
installations, while the Mach site manager needs to manage
the system `sys.path` and conditionally create an on-disk
virtualenv.
By splitting the class into two, we can now give each use case the
attention it deserves.
Differential Revision: https://phabricator.services.mozilla.com/D129529
Sorry for the flip-flop on technique here :S
`validate_environment_packages()` was originally run when checking if a
site is up-to-date to ensure that ad-hoc pip installs didn't replace
needed packages with those of different versions.
However, since it was added, a few notes have come up:
1. The case where requirements change isn't caught by this - that is
caught earlier by the cheap "a requirements file has changed on-disk"
check.
2. This is really slow, and doing it for most Mach commands is not worth
it (as evident by how the `skip_pip_package_check` was already added
for the Mach site's use case).
3. Since the tree as-is doesn't have (common) cases where ad-hoc
installations break an environment, then this check, though helpful,
isn't adding a significant amount of value considering its performance
cost.
However, these aren't to say that this won't be valuable in the future:
I'd like to reach a point where sites are considered "sealed" by
default: no ad-hoc pip installations are allowed.
However, add the ability to mark sites as unsealed/"allowing
ad-hoc pip installations". Then, re-add the pip package check, but only
for such flexible, unsealed virtualenvs.
Differential Revision: https://phabricator.services.mozilla.com/D129692
IIRC, this was added prematurely for an upcoming test due to a faulty
VCS commit split.
Let's temporarily remove this parameter, then re-add it when needed for
whatever test consumes it.
Differential Revision: https://phabricator.services.mozilla.com/D129691
As of bug 1686168, `__PYVENV_LAUNCHER__` is purged when Mach starts
running. Since `MozSiteManager` is only used by Mach, its internal
purging of the environment variable is redundant.
Differential Revision: https://phabricator.services.mozilla.com/D129690
The information about the Python executable is now stored with other
details in the JSON metadata file we put in each virtualenv.
Differential Revision: https://phabricator.services.mozilla.com/D129689
Rather than requiring that consumers remember to `ensure()` before
calling `activate()`, we can do so automatically during activation.
There isn't a valid use case in which we want obsolete virtualenvs to be
activateable.
Usages of `MozSiteManager` have been updated accordingly.
Differential Revision: https://phabricator.services.mozilla.com/D129688
`create()` was only called from one place, and had more complex logic
than necessary.
Related to this change, `check=True` is used rather than asserting the
return code.
Differential Revision: https://phabricator.services.mozilla.com/D129687
`MozVirtualenvMetadata` only needs to know the virtualenv it's operating
on - the knowledge of its internal metadata filename should not have to
be exposed.
Differential Revision: https://phabricator.services.mozilla.com/D129686
The current inheritance-based separation between "general venv
details" (`VirtualenvHelper`) and moz site details
(`MozSiteManager`) has caused `MozSiteManager` to care
about too many details.
This patch splits things apart a little more:
* The term `VirtualenvHelper` isn't _super_ useful, rename it to
`PythonVenv` to show that it represents a Python virtualenv.
* Move on-disk virtualenv logic (`activate_path` location,
`site-packages` location, `pip install` behaviour) into a
contained `MozVirtualenv` class.
* Port the inheritance to a "composition" model instead.
Differential Revision: https://phabricator.services.mozilla.com/D129685
The existing terminology had two issues:
* `VirtualenvManager` wasn't always associated with an on-disk
`virtualenv`: for example, when running in automation, Mach
"activates" a `VirtualenvManager`, updating its import scope,
but without ever creating an on-disk `virtualenv`.
* An upcoming patch splits the `VirtualenvManager` class, pulling
"on-disk virtualenv-handling functions" from the project-wide
interface for managing Python's import scope.
After some good discussion with Ahal, I think we've struck
the terminology that handles this distinction well: we'll call
the "import scope"-handling part the "site", and we'll continue
to call on-disk virtualenvs (and their representative classes)
as, well, virtualenvs.
Differential Revision: https://phabricator.services.mozilla.com/D130391
As `_run_pip()` is being removed from `VirtualenvManager` in an upcoming
patch, its usages need to be removed. Besides, they're using an
"internal" function, which is a bit of a smell.
Note that this _could_ have been solved by exposing a public `run_pip()`
function. However, I felt like that was worse because:
* Friction here is good as we try to migrate the codebase to embrace the
"requirements definition file" technique to install dependencies.
* There could be confusion about the relationship between
`install_pip_package()` (only works if venv already activated)
and `_run_pip()`, which works "in general".
Differential Revision: https://phabricator.services.mozilla.com/D130120
The `base_python` parameter was used when we had both Python 2 and
Python 3 virtualenvs. Since that's no longer the case and it's now
unused, we can remove it.
Differential Revision: https://phabricator.services.mozilla.com/D129299
Now that the "build" virtualenv isn't created during configure, we can
remove the logic that tries to temporarily shelter pip from
configure's changes.
Differential Revision: https://phabricator.services.mozilla.com/D129868
Previously, we always used the last UUID in configure.sh.
This is incorrect for beta because beta and release share the same "official" branding and the beta UUIDs are specified first.
We now decide whether to use the first or last UUID based on the channel.
Differential Revision: https://phabricator.services.mozilla.com/D130833
Unless an upstream is specified, we should be able to detect which files
are changed on our local branch without hitting the network.
Differential Revision: https://phabricator.services.mozilla.com/D130684
We now have more intelligent ways of checking if a venv is out-of-date:
* Are the `pth` files correct?
* Were any requirements files changed?
Differential Revision: https://phabricator.services.mozilla.com/D130678
The virutalenv hack is in the fedora-distributed version of
virtualenv... Presumably eventually will become unnecessary once they
provide a proper "venv" distutils?
This patch applies both before and after the bump in comment 5, so your
call.
Differential Revision: https://phabricator.services.mozilla.com/D130410
It's possible for the `PYTHON3` config to point to a different Python
than that that is executing the configure scripts themselves.
This flexibility was needed for the Python 2->3 migration, but now that
it's complete, we can remove the extra configuration and just lean on
the Python interpreter used to run configure.
Differential Revision: https://phabricator.services.mozilla.com/D129863
There were a bunch of locations where we were doing path shenanigans
with `requirements.pth/.vendored` items.
There was a bit of complexity because we were specifically making each
`pthfile` line be a relative path to support moving the Firefox
topsrcdir without causing issues.
However, now that we're more intelligent about checking if `pthfile`
lines are up-to-date (and since moving your topsrcdir will still require
re-generating the Mach virtualenv), this behaviour became less useful.
So, generalize `MachEnvRequirements` -> "sys.path lines" logic and
reuse it everywhere.
Differential Revision: https://phabricator.services.mozilla.com/D129693
The logic to handle virtualenv requirements definitions now need to be
able to import `packaging` and `pyparsing` in order to parse the pip
requirement specifier notation.
Differential Revision: https://phabricator.services.mozilla.com/D130288
Downloading, as well as resuming a partial download, of an SDK/NDK was
implemented. The progress bar was implemented via tqdm.
This is not a generic solution, but the same ideas could be used and improved
to replace 'dlmanager' in the future.
Differential Revision: https://phabricator.services.mozilla.com/D129063
This patch adds a new command line argument --aab which allows users to install
GVE as an AAB.
This will also be used in a future patch to install the test runner as AAB.
Differential Revision: https://phabricator.services.mozilla.com/D127323
The fact that the test runner app is defined inside the geckoview test package
has always felt like a hack to me. I've mistakenly thought that
TestRunnerActivity was used in GeckoView's junit tests many times (even though
that's not the case).
From what I can see, there's no way to generate an AAB package for androidTest,
so to be able to run Gecko tests as AAB we finally need to define the
TestRunner as an ordinary package instead.
Differential Revision: https://phabricator.services.mozilla.com/D127320
This tool is used to install AAB packages to devices.
AAB is the new package format for Android Apps and will supersede APK.
Differential Revision: https://phabricator.services.mozilla.com/D127318
The optional `log_handle` argument was only used by:
* Configure, but the output was always dumped to stdout and *not*
`config.log`. The manual logging _was_ used to handle encoding issues,
but those are likely invalidated by the Python 3 migration.
* `./mach doc`, where we were putting virtualenv setup into stderr,
which seems incorrect. The commit adding it doesn't explain why it's
the case, but I'm guessing it shouldn't be too risky to remove.
Additionally, `log_handle` was used very inconsistently: for example,
running `install_pip_package()` would _not_ use `log_handle`.
So, removing `log_handle` removes a bit of abstraction leakage.
Differential Revision: https://phabricator.services.mozilla.com/D129298
Mach already verifies the Python version during initialization.
Maintaining a second version check can be tough to keep up-to-date, as
we're already seeing due to the obsolete Python 2 check.
Differential Revision: https://phabricator.services.mozilla.com/D129297
Now that are prioritizing system over virtualenv site-packages, the
system `pip` is sometimes being used instead.
This is causing issues when the system pip is set up in a
distro-specific way, such as when "debundled":
https://github.com/pypa/pip/blob/9.0.1/pip/_vendor/__init__.py#L53-L61
However, if we vendor `pip`, `setuptools` and `wheel`, and ensure that
they're prioritized in the `sys.path` before anything is imported from
the system, then we can ensure that we're using a modern `pip` _and_
sidestep system-specific pip weirdness.
Note that `pip-compile`'s `--allow-unsafe` flag is not as dangerous as
it sounds.
There's confusion among maintainers about its origin:
https://github.com/jazzband/pip-tools/issues/522
Additionally, it's going to be enabled by default in a future
`pip-tools` release. So, it's not scary for us to embrace here.
Also, heads up that the "pip outdated warning" no longer needs
to be manually silenced, since pip avoids that code path when
not running from an "installed" context.
Differential Revision: https://phabricator.services.mozilla.com/D127182
Mitchell Hentges
Tom Ritter
Marian-Vasile Laza
Mike Hommey
Mats Palmgren
Andrew Halberstadt
Iulian Moraru
Sylvestre Ledru
James Teh
Geoff Lankow
Agi Sferro
Csoregi Natalia
Eemeli Aro
Emilio Cobos Álvarez
Alex Hochheiden
Glowreeyah
Sebastian Hengst