зеркало из https://github.com/mozilla/gecko-dev.git
bbfa258584
In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically: 1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change. 2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781. 3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed. As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages. With that in mind, this patch does all the following: 1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it. 2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`. 3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`). 4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3). 5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes. This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes: 1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI. 2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code. Differential Revision: https://phabricator.services.mozilla.com/D85916 |
||
|---|---|---|
| .. | ||
| autoconf | ||
| build-clang | ||
| build-infer | ||
| clang-plugin | ||
| compare-mozconfig | ||
| debian-packages | ||
| docs | ||
| gyp_includes | ||
| liblowercase | ||
| macosx | ||
| mobile | ||
| moz.configure | ||
| package/mac_osx | ||
| pgo | ||
| sanitizers | ||
| sparse-profiles | ||
| tests/cram | ||
| unix | ||
| valgrind | ||
| win32 | ||
| win64 | ||
| win64-aarch64 | ||
| workspace-hack | ||
| .gdbinit | ||
| .gdbinit.loader | ||
| .gdbinit.py.in | ||
| .lldbinit.in | ||
| RunCbindgen.py | ||
| __init__.py | ||
| appini_header.py | ||
| application.ini.in | ||
| binary-location.mk | ||
| build_virtualenv_packages.txt | ||
| buildconfig.py | ||
| cargo-host-linker | ||
| cargo-host-linker.bat | ||
| cargo-linker | ||
| cargo-linker.bat | ||
| checksums.py | ||
| common_virtualenv_packages.txt | ||
| defines.sh | ||
| dumbmake-dependencies | ||
| gecko_templates.mozbuild | ||
| gen_symverscript.py | ||
| gen_test_packages_manifest.py | ||
| genrc.sh | ||
| gn.mozbuild | ||
| gyp.mozbuild | ||
| gyp_base.mozbuild | ||
| mach_bootstrap.py | ||
| mach_virtualenv_packages.txt | ||
| moz-automation.mk | ||
| moz.build | ||
| mozconfig.artifact | ||
| mozconfig.artifact.automation | ||
| mozconfig.automation | ||
| mozconfig.cache | ||
| mozconfig.clang-cl | ||
| mozconfig.comm-support | ||
| mozconfig.common | ||
| mozconfig.common.override | ||
| mozconfig.lld-link | ||
| mozconfig.nasm | ||
| mozconfig.no-compile | ||
| mozconfig.node | ||
| mozconfig.rust | ||
| mozconfig.stylo | ||
| mozconfig.wasm-sandboxing | ||
| mozconfig.win-common | ||
| qemu-wrap | ||
| submit_telemetry_data.py | ||
| templates.mozbuild | ||
| test_templates.mozbuild | ||
| update-settings.ini | ||
| upload.py | ||
| upload_generated_sources.py | ||
| variables.py | ||
| windows_toolchain.py | ||