All commands declaring a virtualenv will have them activated before the
command executes. Removes all now-redundant manual activations of
declared virtualenvs.
Commands that don't declare a virtualenv will still implicitly be
associated with the "common" virtualenv, but unlike explicit
virtualenv declarations it'll have to be activated manually, just
like it was before this patch.
To smooth the migration with existing usages, virtualenv activation
behaviour was changed slightly: if attempting to activate a new
virtualenv, but the source venv is already command venv, then raise an
exception. (In the future, we should improve testability of
virtualenv scaffolding logic so that tests can be added for this
sort of thing.) This did cause some issues with some tests, which
will be solved more cleanly with bug 1724273. In the meantime,
minimal modifications were made to failing tests to keep them green:
* `test_command_line.py` was activating the `common` virtualenv so
that it could install `mozproxy`, and use its CLI. Instead, I
modified the test to use `mozproxy` using the "module" interface
(`python -m mozproxy ...`). At that point, `MozbuildObject` was
unnecessary and usages were replaced with simpler variants.
* `test_vendor.py` needed its explicit `activate_virtualenv()` call
patched out. It still needs to use a virtualenv's Python
executable, but due to `sys.executable` now being kept up-to-date
as of bug 1717051, it could be used directly.
Differential Revision: https://phabricator.services.mozilla.com/D122892
This will give additional feedback to the user. Now they will have
immediate feedback in regards to the progress of the test run, instead
of being unsure of what's happening until the first test run completes.
Differential Revision: https://phabricator.services.mozilla.com/D137925
Since stdout is being piped, redirecting stderr to stdout causes it
to end up in the same pipe interleaved, which is what we want.
Also added a non-zero return code to be set when there is no test output.
Differential Revision: https://phabricator.services.mozilla.com/D138481
There's a bug with `ProcessHandler` on Windows that sometimes causes
tests to hang for an additional 3 minutes. This is a workaround that
just replaces `ProcessHandler` with a standard Python3 approach that
achieves the same thing.
Also added some logging output as the threads are created as some
additional feedback. Prior to this, the tests silently started
running without any indication.
Differential Revision: https://phabricator.services.mozilla.com/D137822
The python-test infra runs each test file in a separate process for
parallelization. This is expedient, but it means that when a test fails, the
output is often buried 1000s of lines in scrollback and difficult to find.
In the future, we could look into using 'pytest-async' rather than rolling our
own parallelization, this way Pytest would output the failures again at the end
of the run.
But for now, it's easier to simply defer outputting failed tests until the very
end. This is done in a try/finally block so even if there was a
KeyboardInterrupt or other exception, we'll still print the failed tests.
Differential Revision: https://phabricator.services.mozilla.com/D135274
`--no-virtualenv` was needed for one use case: the
`get_and_diffoscope` task, despite not needing `psutil`, would
run into failures during virtualenv-creation because we used to
unconditionally build the `psutil` package, and `get_and_diffoscope`
didn't have the environment needed for such a build.
Since we no longer build and install `psutil` into every virtualenv,
it's no longer needed for its one usage, which means that it can be
removed.
`--requirements` is replaced by `--virtualenv`, which removes an
ad-hoc pip package installation and embraces the centralized dep
system.
`--no-activate` is now implied by default: a virtualenv is only
created and activated if `--virtualenv` is provided.`
`ipython==7.16.1` was the chosen version because it is the last
one compatible with Python 3.6.
Differential Revision: https://phabricator.services.mozilla.com/D131529
Rather than requiring that consumers remember to `ensure()` before
calling `activate()`, we can do so automatically during activation.
There isn't a valid use case in which we want obsolete virtualenvs to be
activateable.
Usages of `MozSiteManager` have been updated accordingly.
Differential Revision: https://phabricator.services.mozilla.com/D129688
As `_run_pip()` is being removed from `VirtualenvManager` in an upcoming
patch, its usages need to be removed. Besides, they're using an
"internal" function, which is a bit of a smell.
Note that this _could_ have been solved by exposing a public `run_pip()`
function. However, I felt like that was worse because:
* Friction here is good as we try to migrate the codebase to embrace the
"requirements definition file" technique to install dependencies.
* There could be confusion about the relationship between
`install_pip_package()` (only works if venv already activated)
and `_run_pip()`, which works "in general".
Differential Revision: https://phabricator.services.mozilla.com/D130120
There were a bunch of locations where we were doing path shenanigans
with `requirements.pth/.vendored` items.
There was a bit of complexity because we were specifically making each
`pthfile` line be a relative path to support moving the Firefox
topsrcdir without causing issues.
However, now that we're more intelligent about checking if `pthfile`
lines are up-to-date (and since moving your topsrcdir will still require
re-generating the Mach virtualenv), this behaviour became less useful.
So, generalize `MachEnvRequirements` -> "sys.path lines" logic and
reuse it everywhere.
Differential Revision: https://phabricator.services.mozilla.com/D129693
There's some trade-offs in play here: the major issue is that we can't
pin `psutil`'s because it's pre-installed on some CI workers with a
different version (5.4.2).
Additionally, we can't "just" bump that version, because CI workers jump
between different revisions to do work, so a specific pinned version
won't work when we try to update such a package.
One option is that we could avoid validating package versions in CI, but
then that will cause surprises (heck, I didn't know we were still using
`psutil==5.4.2` instead of `5.8.0` until now). By doing validation, we
make it more explicit and avoid accidentally depending on behaviour of
too new of such a package.
However, in most cases, we manage the installation environment and can
pin dependencies. So, I've made the top-level Mach virtualenv the _only_
one that is able to use requirement specifiers other than "==".
Differential Revision: https://phabricator.services.mozilla.com/D122889
This adds two main compatibility guarantees:
1. Vendored dependencies <=> Pypi-downloaded dependencies
2. Global Mach dependencies <=> command-specific dependencies
As part of this, a new `vendored:` action was added to the virtualenv
definition format. Otherwise similar to `pth:` paths, `vendored:`
packages are assumed to be "pip install"-able.
Some validation (the `.dist-info`/`PKG-INFO` checks) was added to
`requirements.py` to verify that `pth:` and `vendored:` are correctly
used.
Differential Revision: https://phabricator.services.mozilla.com/D122900
Rather than re-implementing it as `search_path()`, use the existing
`MachEnvRequirements` tool to parse `mach_virtualenv_requirements.txt`
Differential Revision: https://phabricator.services.mozilla.com/D126280
There's some trade-offs in play here: the major issue is that we can't
pin `psutil`'s because it's pre-installed on some CI workers with a
different version (5.4.2).
Additionally, we can't "just" bump that version, because CI workers jump
between different revisions to do work, so a specific pinned version
won't work when we try to update such a package.
One option is that we could avoid validating package versions in CI, but
then that will cause surprises (heck, I didn't know we were still using
`psutil==5.4.2` instead of `5.8.0` until now). By doing validation, we
make it more explicit and avoid accidentally depending on behaviour of
too new of such a package.
However, in most cases, we manage the installation environment and can
pin dependencies. So, I've made the top-level Mach virtualenv the _only_
one that is able to use requirement specifiers other than "==".
Differential Revision: https://phabricator.services.mozilla.com/D122889
This adds two main compatibility guarantees:
1. Vendored dependencies <=> Pypi-downloaded dependencies
2. Global Mach dependencies <=> command-specific dependencies
As part of this, a new `vendored:` action was added to the virtualenv
definition format. Otherwise similar to `pth:` paths, `vendored:`
packages are assumed to be "pip install"-able.
Some validation (the `.dist-info`/`PKG-INFO` checks) was added to
`requirements.py` to verify that `pth:` and `vendored:` are correctly
used.
Differential Revision: https://phabricator.services.mozilla.com/D122900
Rather than re-implementing it as `search_path()`, use the existing
`MachEnvRequirements` tool to parse `mach_virtualenv_requirements.txt`
Differential Revision: https://phabricator.services.mozilla.com/D126280
This removes the `@CommandProvider` decorator and the need to implement
mach commands inside subclasses of `MachCommandBase`, and moves all
existing commands out from classes to module level functions.
Differential Revision: https://phabricator.services.mozilla.com/D121512
This removes the `@CommandProvider` decorator and the need to implement
mach commands inside subclasses of `MachCommandBase`, and moves all
existing commands out from classes to module level functions.
Differential Revision: https://phabricator.services.mozilla.com/D121512
This removes the `@CommandProvider` decorator and the need to implement
mach commands inside subclasses of `MachCommandBase`, and moves all
existing commands out from classes to module level functions.
Differential Revision: https://phabricator.services.mozilla.com/D121512
This removes the `@CommandProvider` decorator and the need to implement
mach commands inside subclasses of `MachCommandBase`, and moves all
existing commands out from classes to module level functions.
Differential Revision: https://phabricator.services.mozilla.com/D121512
We've overloaded "bootstrap" to mean three different things:
* The "standalone bootstrap script": `python/mozboot/bin/bootstrap.py`.
This is to freshly clone a new repo, then run `./mach bootstrap`.
* `./mach bootstrap`: Install necessary dependencies and set up the
system for development.
* "Mach bootstrap": do the in-process initialization work Mach needs
before it can run commands.
By using the term "initialize" instead, perhaps we can remove
ambiguity when discussing Mach.
I'm not attached to the name (or this change at all), but I'm interested
in reviewer thoughts :)
Differential Revision: https://phabricator.services.mozilla.com/D120410
This step removes all the dependencies of mach commands to
having a MachCommandBase as the `self` by using the `command_context`
argument instead. This also removes any remaining statefulness from those
classes that implement mach commands, ultimately making it easier to move
existing commands out of classes in a follow-up.
Differential Revision: https://phabricator.services.mozilla.com/D118058
This step removes all the dependencies of mach commands to
having a MachCommandBase as the `self` by using the `command_context`
argument instead. This also removes any remaining statefulness from those
classes that implement mach commands, ultimately making it easier to move
existing commands out of classes in a follow-up.
Differential Revision: https://phabricator.services.mozilla.com/D118058
After some testing in `try` and locally, the manual
`PYTHONEXECUTABLE` definitions shouldn't be needed
anymore.
There's been some work on Brew's
Python to improve its `sitecustomize` behaviour.
The most likely improvement is:
https://github.com/Homebrew/homebrew-core/pull/65297
However, I'm not fully confident in this change.
If it fails, it's more likely to affect developers
than CI. I think it's worth attempting a landing,
because if the variable is indeed obsolete, then
deleting it will avoid some spicy "action-at-a-
distance" behaviour.
Differential Revision: https://phabricator.services.mozilla.com/D117452
At least in PyCharm, expanding a `VirtualenvManager` instance means
resolving all the properties and fields of the instance.
However, if that property is doing non-trivial work, the debugger
wouldn't run that subprocess while we're stopped at a breakpoint.
So, the instance would sit there with the "Collecting data..." text.
Differential Revision: https://phabricator.services.mozilla.com/D115935
At least in PyCharm, expanding a `VirtualenvManager` instance means
resolving all the properties and fields of the instance.
However, if that property is doing non-trivial work, the debugger
wouldn't run that subprocess while we're stopped at a breakpoint.
So, the instance would sit there with the "Collecting data..." text.
Differential Revision: https://phabricator.services.mozilla.com/D115935
As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).
Differential Revision: https://phabricator.services.mozilla.com/D109650
As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).
Differential Revision: https://phabricator.services.mozilla.com/D109650
As an intermediate step to allow mach commands as standalone functions, the MachCommandBase
subclass instance that currently corresponds to self has to be made available as a separate
argument (named command_context).
Differential Revision: https://phabricator.services.mozilla.com/D109650
Now that all the conflicts in python dependencies that made the
new pip resolver fail have been resolved, we no longer need to support
the legacy resolver that was added as a temporary measure.
Depends on D106154
Differential Revision: https://phabricator.services.mozilla.com/D106503
pylint_requirements.txt fail to install with the new pip resolver due
to a conflict between astroid and lazy-object-proxy.
Rather than bumping those packages and handling the potential fallout,
the package-upgrade has been deferred and we will use the legacy
resolver in the interrim.
Differential Revision: https://phabricator.services.mozilla.com/D99940
pylint_requirements.txt fail to install with the new pip resolver due
to a conflict between astroid and lazy-object-proxy.
Rather than bumping those packages and handling the potential fallout,
the package-upgrade has been deferred and we will use the legacy
resolver in the interrim.
Differential Revision: https://phabricator.services.mozilla.com/D99940
CLOSED TREE
Backed out changeset baa1f7a615e4 (bug 1666345)
Backed out changeset b6646baa866d (bug 1661624)
Backed out changeset e4d550db6037 (bug 1667152)
Mitchell Hentges
ahochheiden
Andrew Halberstadt
criss
Alex Lopez
Butkovits Atila
Noemi Erli
Dorel Luca
Cosmin Sabou
Ricky Stewart
Mihai Alexandru Michis
smolnar