CLOSED TREE
Backed out changeset baa1f7a615e4 (bug 1666345)
Backed out changeset b6646baa866d (bug 1661624)
Backed out changeset e4d550db6037 (bug 1667152)
With PYTHONPATH containing other directories, such as the python ones,
bad things can happen when the python script that is being run then
goes on to subprocess.Popen a python process for a different virtualenv,
or a different version, or whatever.
Differential Revision: https://phabricator.services.mozilla.com/D96155
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
If you set a `DEFAULT` `requirements` file in a Python test manifest, the current implementation of `python-test` will try (and usually fail) to install the file once for every test file in parallel. Instead, initializing the environment should be done once when the command starts up.
Differential Revision: https://phabricator.services.mozilla.com/D90475
Pipenv is heavy weight and overkill for the purposes it is being used. We'd like to remove it from the tree and |mach python-test| was one of the last remanining use cases.
Remove the `--python` command-line argument as a result. Users who wish to run unit tests with Python 2 can do `MACH_PY2=1 ./mach python-test ...` or `python2 ./mach python-test ...`.
Also update a few unit tests that would break otherwise in the presence of this change.
There were a couple lines in the `setup.py` for `mozlog` that were problematic for tests and was resulting in errors due to the `mozlog` plugin being loaded by `pytest` more than once. We just delete those lines and bump up the major version number of the package to fix it.
Differential Revision: https://phabricator.services.mozilla.com/D88296
To ensure that a python 3 virtualenv exists for tests, python-test will ensure that it's created (regardless of which version of python is currently in use).
However, the existing logic was incorrectly creating an "extra py3 virtualenv", even if currently running python 3 and having already created a "first" py3 virtualenv
Differential Revision: https://phabricator.services.mozilla.com/D86269
I noticed that the `objdir:build` entry in `build/virtualenv_packages.txt` entry was apparently unused. This originates from bug 841713, seven years ago, when the `objdir` handling was introduced. Today, this doesn't appear to be serving a purpose. There is no Python library in my `$objdir/build` directory; nor can I see anything in `build/moz.build` or any related files suggesting one could ever appear. I can only assume this feature has outlived its usefulness, so delete it and the relevant in-tree support.
This necessitates slightly changing the signature and implementation of the `activate_pipenv()` method; also update all callers.
Differential Revision: https://phabricator.services.mozilla.com/D85635
I noticed that the `objdir:build` entry in `build/virtualenv_packages.txt` entry was apparently unused. This originates from bug 841713, seven years ago, when the `objdir` handling was introduced. Today, this doesn't appear to be serving a purpose. There is no Python library in my `$objdir/build` directory; nor can I see anything in `build/moz.build` or any related files suggesting one could ever appear. I can only assume this feature has outlived its usefulness, so delete it and the relevant in-tree support.
This necessitates slightly changing the signature and implementation of the `activate_pipenv()` method; also update all callers.
Differential Revision: https://phabricator.services.mozilla.com/D85635
The existence of this environment variable breaks the Python shell on Windows, so make sure it's unset (but only in this case to avoid regressing bug 1627873).
Differential Revision: https://phabricator.services.mozilla.com/D70542
We recently updated the version of pip we use, and it no longer supports
--use-wheels.
Differential Revision: https://phabricator.services.mozilla.com/D57404
--HG--
extra : moz-landing-system : lando
Make the $PYTHON3 build var point to a full virtualenv bootstrapped with
the same libraries as the $PYTHON Python 2 build var. This allows us to
upgrade build tasks from $PYTHON to $PYTHON3.
This patch adds some debug logging and documentation to the Python
2 virtualenv so that it is easier to diagnose issues that may arise
from running two different Python interpreters in re-entrant
multiprocess routines.
Differential Revision: https://phabricator.services.mozilla.com/D50819
--HG--
extra : moz-landing-system : lando
Make the $PYTHON3 build var point to a full virtualenv bootstrapped with
the same libraries as the $PYTHON Python 2 build var. This allows us to
upgrade build tasks from $PYTHON to $PYTHON3.
This patch adds some debug logging and documentation to the Python
2 virtualenv so that it is easier to diagnose issues that may arise
from running two different Python interpreters in re-entrant
multiprocess routines.
Differential Revision: https://phabricator.services.mozilla.com/D50819
--HG--
extra : moz-landing-system : lando
Homebrew on OS X will change Python's sys.executable to a custom value
which messes with mach's virtualenv handling code. Override Homebrew's
changes with the correct sys.executable value.
Differential Revision: https://phabricator.services.mozilla.com/D54602
--HG--
extra : moz-landing-system : lando
This will install ipython into the default virtualenv if it doesn't exist. Unless --no-virtualenv
is specified in which case an error will be printed.
Differential Revision: https://phabricator.services.mozilla.com/D53030
--HG--
extra : moz-landing-system : lando
Sometimes tools install pypi at runtime via mach (e.g self.install_pip_package
/ self.install_pip_requirements). It's difficult to test these modules with
pytest because we usually won't be going through mach.
This gives tests the ability to depend on external pypi packages the same way
they might get installed when running via mach.
Note, I only added support for requirements.txt here because
python/mozbuild/mozbuild/virtualenv.py's 'install_pip_package' function is
completely busted with modern pip. And the pip used with |mach python-test| is
more modern than the one used with the regular build venv due to pipenv. We'll
need to fix this eventually, but that's another bug for another day.
Differential Revision: https://phabricator.services.mozilla.com/D22784
--HG--
extra : moz-landing-system : lando
This change tries to ensure that we don't write telemetry data for mach
commands invoked recursively as part of other mach commands. The intent of
build system telemetry is to only collect data about commands that users are
invoking directly.
There are two ways that we found mach commands can be recursively invoked:
* By running a python subprocess to recursively invoke mach (used in
`mach bootstrap` to call `mach artifact toolchain`)
* By using `Registrar.dispatch` to delegate to a sub-command (used by many
build system commands to invoke `mach build`).
The subprocess case is handled here by having mach set a `MACH_MAIN_PID`
environment variable whose value is the current process' pid on startup if it
does not already exist in the environment. Telemetry code then checks that the
value of that variable matches the current pid and skips writing telemetry data
if not.
The dispatch case is handled by making `MachRegistrar` store the current depth
of the command stack and pass it to the `post_dispatch_handler` which will skip
writing telemetry data if depth != 1.
Additionally the `should_skip_dispatch` function in mach_bootstrap is renamed
to `should_skip_telemetry_submission`, which was its original intent. The
combination of checks added in this change should be sufficient for deciding
when to write telemetry data, and we were not collecting telemetry for the set
of mach commands in that function (which included `mach bootstrap`).
In order to facilitate writing a test for the dispatch case this change adds a
`mach python --exec-file` option to execute Python code directly in the context
of the `mach python` command.
Differential Revision: https://phabricator.services.mozilla.com/D11207
--HG--
extra : moz-landing-system : lando
This patch uses the PIPENV_PYTHON environment variable to append a suffix to the created virtual environment path according to the version specified. It also uses the PIPENV_DEFAULT_PYTHON_VERSION environment variable to avoid recreating the virtual environment every time. With these changes we are able to switch back and forth between Python versions without the expense of recreating environments, however there is a risk of these environments becoming stale. In this scenario it may be necessary to clobber the virtual environment root within the obj dir.
MozReview-Commit-ID: C4vuwNh04CP
--HG--
extra : rebase_source : 301c8418fe5b873ffadec37af11e98cfda8667b8
This patch uses the PIPENV_PYTHON environment variable to append a suffix to the created virtual environment path according to the version specified. It also uses the PIPENV_DEFAULT_PYTHON_VERSION environment variable to avoid recreating the virtual environment every time. With these changes we are able to switch back and forth between Python versions without the expense of recreating environments, however there is a risk of these environments becoming stale. In this scenario it may be necessary to clobber the virtual environment root within the obj dir.
MozReview-Commit-ID: C4vuwNh04CP
--HG--
extra : rebase_source : ba34e415fa21683bc2a39231651587fa30ad0242
Ricky Stewart
Noemi Erli
Mihai Alexandru Michis
smolnar
Mike Hommey
Bogdan Tara
Dorel Luca
Mitchell Hentges
Hamzah
Andrew Halberstadt
Csoregi Natalia
Māris Fogels
Sebastian Hengst
Anmol Agarwal
Henri Sivonen
Henrik Skupin
Ted Mielczarek
Dave Hunt
Andreea Pavel