`--no-virtualenv` was needed for one use case: the
`get_and_diffoscope` task, despite not needing `psutil`, would
run into failures during virtualenv-creation because we used to
unconditionally build the `psutil` package, and `get_and_diffoscope`
didn't have the environment needed for such a build.
Since we no longer build and install `psutil` into every virtualenv,
it's no longer needed for its one usage, which means that it can be
removed.
`--requirements` is replaced by `--virtualenv`, which removes an
ad-hoc pip package installation and embraces the centralized dep
system.
`--no-activate` is now implied by default: a virtualenv is only
created and activated if `--virtualenv` is provided.`
`ipython==7.16.1` was the chosen version because it is the last
one compatible with Python 3.6.
Differential Revision: https://phabricator.services.mozilla.com/D131529
Rather than requiring that consumers remember to `ensure()` before
calling `activate()`, we can do so automatically during activation.
There isn't a valid use case in which we want obsolete virtualenvs to be
activateable.
Usages of `MozSiteManager` have been updated accordingly.
Differential Revision: https://phabricator.services.mozilla.com/D129688
As `_run_pip()` is being removed from `VirtualenvManager` in an upcoming
patch, its usages need to be removed. Besides, they're using an
"internal" function, which is a bit of a smell.
Note that this _could_ have been solved by exposing a public `run_pip()`
function. However, I felt like that was worse because:
* Friction here is good as we try to migrate the codebase to embrace the
"requirements definition file" technique to install dependencies.
* There could be confusion about the relationship between
`install_pip_package()` (only works if venv already activated)
and `_run_pip()`, which works "in general".
Differential Revision: https://phabricator.services.mozilla.com/D130120
There were a bunch of locations where we were doing path shenanigans
with `requirements.pth/.vendored` items.
There was a bit of complexity because we were specifically making each
`pthfile` line be a relative path to support moving the Firefox
topsrcdir without causing issues.
However, now that we're more intelligent about checking if `pthfile`
lines are up-to-date (and since moving your topsrcdir will still require
re-generating the Mach virtualenv), this behaviour became less useful.
So, generalize `MachEnvRequirements` -> "sys.path lines" logic and
reuse it everywhere.
Differential Revision: https://phabricator.services.mozilla.com/D129693
There's some trade-offs in play here: the major issue is that we can't
pin `psutil`'s because it's pre-installed on some CI workers with a
different version (5.4.2).
Additionally, we can't "just" bump that version, because CI workers jump
between different revisions to do work, so a specific pinned version
won't work when we try to update such a package.
One option is that we could avoid validating package versions in CI, but
then that will cause surprises (heck, I didn't know we were still using
`psutil==5.4.2` instead of `5.8.0` until now). By doing validation, we
make it more explicit and avoid accidentally depending on behaviour of
too new of such a package.
However, in most cases, we manage the installation environment and can
pin dependencies. So, I've made the top-level Mach virtualenv the _only_
one that is able to use requirement specifiers other than "==".
Differential Revision: https://phabricator.services.mozilla.com/D122889
This adds two main compatibility guarantees:
1. Vendored dependencies <=> Pypi-downloaded dependencies
2. Global Mach dependencies <=> command-specific dependencies
As part of this, a new `vendored:` action was added to the virtualenv
definition format. Otherwise similar to `pth:` paths, `vendored:`
packages are assumed to be "pip install"-able.
Some validation (the `.dist-info`/`PKG-INFO` checks) was added to
`requirements.py` to verify that `pth:` and `vendored:` are correctly
used.
Differential Revision: https://phabricator.services.mozilla.com/D122900
Rather than re-implementing it as `search_path()`, use the existing
`MachEnvRequirements` tool to parse `mach_virtualenv_requirements.txt`
Differential Revision: https://phabricator.services.mozilla.com/D126280
There's some trade-offs in play here: the major issue is that we can't
pin `psutil`'s because it's pre-installed on some CI workers with a
different version (5.4.2).
Additionally, we can't "just" bump that version, because CI workers jump
between different revisions to do work, so a specific pinned version
won't work when we try to update such a package.
One option is that we could avoid validating package versions in CI, but
then that will cause surprises (heck, I didn't know we were still using
`psutil==5.4.2` instead of `5.8.0` until now). By doing validation, we
make it more explicit and avoid accidentally depending on behaviour of
too new of such a package.
However, in most cases, we manage the installation environment and can
pin dependencies. So, I've made the top-level Mach virtualenv the _only_
one that is able to use requirement specifiers other than "==".
Differential Revision: https://phabricator.services.mozilla.com/D122889
This adds two main compatibility guarantees:
1. Vendored dependencies <=> Pypi-downloaded dependencies
2. Global Mach dependencies <=> command-specific dependencies
As part of this, a new `vendored:` action was added to the virtualenv
definition format. Otherwise similar to `pth:` paths, `vendored:`
packages are assumed to be "pip install"-able.
Some validation (the `.dist-info`/`PKG-INFO` checks) was added to
`requirements.py` to verify that `pth:` and `vendored:` are correctly
used.
Differential Revision: https://phabricator.services.mozilla.com/D122900
Rather than re-implementing it as `search_path()`, use the existing
`MachEnvRequirements` tool to parse `mach_virtualenv_requirements.txt`
Differential Revision: https://phabricator.services.mozilla.com/D126280
This removes the `@CommandProvider` decorator and the need to implement
mach commands inside subclasses of `MachCommandBase`, and moves all
existing commands out from classes to module level functions.
Differential Revision: https://phabricator.services.mozilla.com/D121512
This removes the `@CommandProvider` decorator and the need to implement
mach commands inside subclasses of `MachCommandBase`, and moves all
existing commands out from classes to module level functions.
Differential Revision: https://phabricator.services.mozilla.com/D121512
This removes the `@CommandProvider` decorator and the need to implement
mach commands inside subclasses of `MachCommandBase`, and moves all
existing commands out from classes to module level functions.
Differential Revision: https://phabricator.services.mozilla.com/D121512
This removes the `@CommandProvider` decorator and the need to implement
mach commands inside subclasses of `MachCommandBase`, and moves all
existing commands out from classes to module level functions.
Differential Revision: https://phabricator.services.mozilla.com/D121512
We've overloaded "bootstrap" to mean three different things:
* The "standalone bootstrap script": `python/mozboot/bin/bootstrap.py`.
This is to freshly clone a new repo, then run `./mach bootstrap`.
* `./mach bootstrap`: Install necessary dependencies and set up the
system for development.
* "Mach bootstrap": do the in-process initialization work Mach needs
before it can run commands.
By using the term "initialize" instead, perhaps we can remove
ambiguity when discussing Mach.
I'm not attached to the name (or this change at all), but I'm interested
in reviewer thoughts :)
Differential Revision: https://phabricator.services.mozilla.com/D120410
This step removes all the dependencies of mach commands to
having a MachCommandBase as the `self` by using the `command_context`
argument instead. This also removes any remaining statefulness from those
classes that implement mach commands, ultimately making it easier to move
existing commands out of classes in a follow-up.
Differential Revision: https://phabricator.services.mozilla.com/D118058
This step removes all the dependencies of mach commands to
having a MachCommandBase as the `self` by using the `command_context`
argument instead. This also removes any remaining statefulness from those
classes that implement mach commands, ultimately making it easier to move
existing commands out of classes in a follow-up.
Differential Revision: https://phabricator.services.mozilla.com/D118058
After some testing in `try` and locally, the manual
`PYTHONEXECUTABLE` definitions shouldn't be needed
anymore.
There's been some work on Brew's
Python to improve its `sitecustomize` behaviour.
The most likely improvement is:
https://github.com/Homebrew/homebrew-core/pull/65297
However, I'm not fully confident in this change.
If it fails, it's more likely to affect developers
than CI. I think it's worth attempting a landing,
because if the variable is indeed obsolete, then
deleting it will avoid some spicy "action-at-a-
distance" behaviour.
Differential Revision: https://phabricator.services.mozilla.com/D117452
At least in PyCharm, expanding a `VirtualenvManager` instance means
resolving all the properties and fields of the instance.
However, if that property is doing non-trivial work, the debugger
wouldn't run that subprocess while we're stopped at a breakpoint.
So, the instance would sit there with the "Collecting data..." text.
Differential Revision: https://phabricator.services.mozilla.com/D115935
At least in PyCharm, expanding a `VirtualenvManager` instance means
resolving all the properties and fields of the instance.
However, if that property is doing non-trivial work, the debugger
wouldn't run that subprocess while we're stopped at a breakpoint.
So, the instance would sit there with the "Collecting data..." text.
Differential Revision: https://phabricator.services.mozilla.com/D115935
As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).
Differential Revision: https://phabricator.services.mozilla.com/D109650
As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).
Differential Revision: https://phabricator.services.mozilla.com/D109650
As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).
Differential Revision: https://phabricator.services.mozilla.com/D109650
Now that all the conflicts in python dependencies that made the
new pip resolver fail have been resolved, we no longer need to support
the legacy resolver that was added as a temporary measure.
Depends on D106154
Differential Revision: https://phabricator.services.mozilla.com/D106503
pylint_requirements.txt fail to install with the new pip resolver due
to a conflict between astroid and lazy-object-proxy.
Rather than bumping those packages and handling the potential fallout,
the package-upgrade has been deferred and we will use the legacy
resolver in the interrim.
Differential Revision: https://phabricator.services.mozilla.com/D99940
pylint_requirements.txt fail to install with the new pip resolver due
to a conflict between astroid and lazy-object-proxy.
Rather than bumping those packages and handling the potential fallout,
the package-upgrade has been deferred and we will use the legacy
resolver in the interrim.
Differential Revision: https://phabricator.services.mozilla.com/D99940
CLOSED TREE
Backed out changeset baa1f7a615e4 (bug 1666345)
Backed out changeset b6646baa866d (bug 1661624)
Backed out changeset e4d550db6037 (bug 1667152)
With PYTHONPATH containing other directories, such as the python ones,
bad things can happen when the python script that is being run then
goes on to subprocess.Popen a python process for a different virtualenv,
or a different version, or whatever.
Differential Revision: https://phabricator.services.mozilla.com/D96155
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045