gecko-dev/build/mach_virtualenv_packages.txt

packages.txt:build/common_virtualenv_packages.txt
# glean-sdk may not be installable if a wheel isn't available
# and it has to be built from source.
pypi-optional:glean-sdk==40.0.0:telemetry will not be collected
# Mach gracefully handles the case where `psutil` is unavailable.
# We aren't (yet) able to pin packages in automation, so we have to
# support down to the oldest locally-installed version (5.4.2).
pypi-optional:psutil>=5.4.2,<=5.8.0:telemetry will be missing some data
pypi-optional:zstandard>=0.11.1,<=0.16.0:zstd archives will not be possible to extract
Bug 1656993: Create and require by default global `virtualenv`s in `~/.mozbuild` for `mach` r=mhentges,ahal In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically: 1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change. 2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781. 3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed. As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages. With that in mind, this patch does all the following: 1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it. 2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`. 3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`). 4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3). 5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes. This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes: 1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI. 2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code. Differential Revision: https://phabricator.services.mozilla.com/D85916 2020-08-17 20:21:02 +03:00			`packages.txt:build/common_virtualenv_packages.txt`
Bug 1717051: Spell `glean-sdk` with a dash, not an underscore r=ahal The Glean SDK was traditionally represented in-tree as `glean_sdk`. However, on PyPI and in `pip list`, it appears as `glean-sdk` (`pip` was doing the translation implicitly and internally). This is breaking our virtualenv state-checking code, since it thinks that `glean_sdk` isn't installed. Since `glean_sdk` was marked as `pypi-optional`, this hasn't affected us yet, but it will when we bump the Glean SDK version, because developers' virtualenvs won't be detected as out-of-date. Differential Revision: https://phabricator.services.mozilla.com/D122500 2021-08-13 20:50:13 +03:00			`# glean-sdk may not be installable if a wheel isn't available`
Bug 1717051: Move global mach dependencies to requirements definition r=ahal After removing `optional` in Bug 1712804, we need to add a variant back here because there's fallible dependencies. However, I've tweaked the re-introduction of the feature to require a specific repercussion message as well. This seemed like a decent tradeoff - the developer becomes aware that the failure is bad, it has repercussions, but it's not a blocking issue. Additionally, since we're printing pip's output, the developer will be able to see the underlying error causing the warning. I also added comment functionality to requirements definitions to allow adjacent documentation of why some requirements are fallible. (Related: I'm looking forward to `mach_bootstrap` not needing to parse requirements definitions. Almost there!) Note that we'll temporarily lose the "pinned" nature of the three moved dependencies until dependency locking is implemented for Mach requirements definitions. Also note that the pinned `zstandard_requirements.txt` can't be removed like the other files because it still has a dangling usage. Finally, in preparation for review: I didn't make `PypiOptionalSpecifier` extend `PypiSpecifier` because I figured that the benefit of flexibility (easier to allow implementations to diverge without needing to untangle an inheritance relationship) was larger than the cost of needing to add properties to both specifiers. If we wanted re-use, I'd probably have `PypiOptionalSpecifier` _contain_ a `PypiSpecifier`, but then you have to reach deeper into the object to get data, so shrug. Differential Revision: https://phabricator.services.mozilla.com/D119835 2021-08-05 18:14:20 +03:00			`# and it has to be built from source.`
Bug 1712151: Bump glean-sdk version to 40.0.0 r=ahal We vendor `glean_parser==3.6.0`, and that was incompatible with `glean-sdk==36.0.0`'s requirement of `glean_parser==2.5.0`. `glean-sdk==40.0.0` expects `glean_parser==3.6.0`, which is perfect. Differential Revision: https://phabricator.services.mozilla.com/D122899 2021-09-03 18:22:21 +03:00			`pypi-optional:glean-sdk==40.0.0:telemetry will not be collected`
Bug 1717051: Move global mach dependencies to requirements definition r=ahal After removing `optional` in Bug 1712804, we need to add a variant back here because there's fallible dependencies. However, I've tweaked the re-introduction of the feature to require a specific repercussion message as well. This seemed like a decent tradeoff - the developer becomes aware that the failure is bad, it has repercussions, but it's not a blocking issue. Additionally, since we're printing pip's output, the developer will be able to see the underlying error causing the warning. I also added comment functionality to requirements definitions to allow adjacent documentation of why some requirements are fallible. (Related: I'm looking forward to `mach_bootstrap` not needing to parse requirements definitions. Almost there!) Note that we'll temporarily lose the "pinned" nature of the three moved dependencies until dependency locking is implemented for Mach requirements definitions. Also note that the pinned `zstandard_requirements.txt` can't be removed like the other files because it still has a dangling usage. Finally, in preparation for review: I didn't make `PypiOptionalSpecifier` extend `PypiSpecifier` because I figured that the benefit of flexibility (easier to allow implementations to diverge without needing to untangle an inheritance relationship) was larger than the cost of needing to add properties to both specifiers. If we wanted re-use, I'd probably have `PypiOptionalSpecifier` _contain_ a `PypiSpecifier`, but then you have to reach deeper into the object to get data, so shrug. Differential Revision: https://phabricator.services.mozilla.com/D119835 2021-08-05 18:14:20 +03:00			# Mach gracefully handles the case where `psutil` is unavailable.
Bug 1723031: Allow flexible dependency-specification in the Mach venv r=ahal There's some trade-offs in play here: the major issue is that we can't pin `psutil`'s because it's pre-installed on some CI workers with a different version (5.4.2). Additionally, we can't "just" bump that version, because CI workers jump between different revisions to do work, so a specific pinned version won't work when we try to update such a package. One option is that we could avoid validating package versions in CI, but then that will cause surprises (heck, I didn't know we were still using `psutil==5.4.2` instead of `5.8.0` until now). By doing validation, we make it more explicit and avoid accidentally depending on behaviour of too new of such a package. However, in most cases, we manage the installation environment and can pin dependencies. So, I've made the top-level Mach virtualenv the _only_ one that is able to use requirement specifiers other than "==". Differential Revision: https://phabricator.services.mozilla.com/D122889 2021-09-28 17:59:29 +03:00			`# We aren't (yet) able to pin packages in automation, so we have to`
			`# support down to the oldest locally-installed version (5.4.2).`
			`pypi-optional:psutil>=5.4.2,<=5.8.0:telemetry will be missing some data`
Bug 1736458 - Upgrade python-zstandard to 0.16.0. r=firefox-build-system-reviewers,andi Differential Revision: https://phabricator.services.mozilla.com/D128807 2021-10-19 10:00:36 +03:00			`pypi-optional:zstandard>=0.11.1,<=0.16.0:zstd archives will not be possible to extract`