Unfortunately, the dependencies of l10nregistry-rs require tokio 1.0. This
allows this third duplicated instance. I think it makes more sense to try to
move the other uses in tree to a more modern version than to try to downgrade
the version of tokio required by l10nregistry-rs and dependencies.
Differential Revision: https://phabricator.services.mozilla.com/D139854
The dataReportingNotification is the last place where the .properties
vendorShortName is used. As notifications already support Fluent,
porting this to use Fluent as well seems pretty strqaightforward.
Differential Revision: https://phabricator.services.mozilla.com/D141860
The Mach site is asserted to be compatible with all command sites, but
command sites are deliberately //not// asserted to be compatible with
each other - they're //supposed// to have the flexibility of being
able to be incompatible.
Accordingly, let's fail loudly if code tries to activate from one
command site to another.
Note that, due to the Mach site being a superset of common, it's safe to
activate from "common" to "<other command site>". This is needed for
cases like `./mach python --virtualenv ...`.
This required updating the `mozproxy` tests, who would deliberately
activate the `common` site so they could call a `mozproxy` entry point
script. These tests were fixed by instead invoking `mozproxy` as a
module (`-m`) of the current `python-test` site.
Differential Revision: https://phabricator.services.mozilla.com/D141659
This is needed for a few reasons:
* All mach commands can use virtualenvs, not just `build`-related
commands, so the files don't make sense to be in `build/`.
* When locking is added, more files associated with virtualenvs will be
added, and this will change will ease the related directory structure
setup.
* This removes the need for a redundant "_virtualenv_packages" keyword
as part of the manifest filenames.
Differential Revision: https://phabricator.services.mozilla.com/D140382
The intention is to remove all the files in the vendoring directory
(which may be the yaml directory or may not be) except those
specified in 'keep'.
However convert_patterns_to_paths expects a list of patterns, and
we were providing it with only one. This resulted in it iterating
over '**' as a string (one iteration for '*' and a second for '*')
This prevented it from traversing recursively. Turning it into a
list fixes it.
Depends on D141901
Differential Revision: https://phabricator.services.mozilla.com/D141902
This creates a way to override the 'tracking: tag' mechnaism
of a moz.yaml file without having to edit it. Useful to
test a library update when there is no new tag available.
Differential Revision: https://phabricator.services.mozilla.com/D141901
Update DSN for reporting exceptions to the new Sentry instance, and
update CI job to create releases on the new Sentry as well.
Differential Revision: https://phabricator.services.mozilla.com/D141122
Since bug 1717104 changed `./mach vendor python` to no longer have a
manual virtualenv activation, we can remove the test mocking that
softly handled the activation.
Differential Revision: https://phabricator.services.mozilla.com/D141674
All commands declaring a virtualenv will have them activated before the
command executes. Removes all now-redundant manual activations of
declared virtualenvs.
Commands that don't declare a virtualenv will still implicitly be
associated with the "common" virtualenv, but unlike explicit
virtualenv declarations it'll have to be activated manually, just
like it was before this patch.
To smooth the migration with existing usages, virtualenv activation
behaviour was changed slightly: if attempting to activate a new
virtualenv, but the source venv is already command venv, then raise an
exception. (In the future, we should improve testability of
virtualenv scaffolding logic so that tests can be added for this
sort of thing.) This did cause some issues with some tests, which
will be solved more cleanly with bug 1724273. In the meantime,
minimal modifications were made to failing tests to keep them green:
* `test_command_line.py` was activating the `common` virtualenv so
that it could install `mozproxy`, and use its CLI. Instead, I
modified the test to use `mozproxy` using the "module" interface
(`python -m mozproxy ...`). At that point, `MozbuildObject` was
unnecessary and usages were replaced with simpler variants.
* `test_vendor.py` needed its explicit `activate_virtualenv()` call
patched out. It still needs to use a virtualenv's Python
executable, but due to `sys.executable` now being kept up-to-date
as of bug 1717051, it could be used directly.
Differential Revision: https://phabricator.services.mozilla.com/D122892
Creates/updates virtualenvs for some mach commands, replacing
their ad-hoc usage of `install_pip_package()`, `pip install`,
and `sys.path` modifications.
Note: The `docs` virtualenv has `Sphinx==1.1.3` installed, even
though a more modern version of `Sphinx` is used when
`./mach doc` is run. This is ok for now, since `./mach doc` will
just install the newer `Sphinx` over top of the old one. Secondarily,
when we port `./mach doc` to use the centralized system, we'll
be incentivized to make the different `doc` commands use synchonized
versions of the same packages. Success!
Also, note that manual installation of `html5lib` and `requests`
isn't ported to the `wpt` site: this is because they're already
provided by the inherited Mach site.
Differential Revision: https://phabricator.services.mozilla.com/D122902
There are cases in CI where sites are being considered out-of-date,
despite the situation being unexpected - for example, the
"site out-of-date even though already activated" issue described in
this patch's associated bug.
Return a `reason` string from `_is_venv_up_to_date()`, and print it in
cases where a virtualenv being out-of-date is erroneous.
Differential Revision: https://phabricator.services.mozilla.com/D140855
The Mach site management has been significantly simplified
in `mach_initialize.py`, so less of the `MachSiteManager`
API is used. Privatize such now-externally-unused functions,
and no longer return unneeded values.
Differential Revision: https://phabricator.services.mozilla.com/D140854
There are cases in CI where sites are being considered out-of-date,
despite the situation being unexpected - for example, the
"site out-of-date even though already activated" issue described in
this patch's associated bug.
Return a `reason` string from `_is_venv_up_to_date()`, and print it in
cases where a virtualenv being out-of-date is erroneous.
Differential Revision: https://phabricator.services.mozilla.com/D140855
The Mach site management has been significantly simplified
in `mach_initialize.py`, so less of the `MachSiteManager`
API is used. Privatize such now-externally-unused functions,
and no longer return unneeded values.
Differential Revision: https://phabricator.services.mozilla.com/D140854
Previously, when trying to determine the minimal `sys.path` that just
includes the standard library (but no `pip`-installed packages), we
would resolve the `sys.path` and remove the system and user
site-packages. However, this "removal" step didn't work as-is for
`brew`'s Python, because its "system site-packages" is //different//
from its `site.getsitepackages()`.
Stepping back though, there's an easier solution: run `python` with the
`-S` flag, and no "site" paths or custom initialization logic will add
anything extra to the stdlib.
Differential Revision: https://phabricator.services.mozilla.com/D141508
Now that we're using `sysconfig` instead of `distutils`, our `purelib`
and `platlib` paths should correctly point to python environment paths
without a spurious `local/` component at their roots.
Differential Revision: https://phabricator.services.mozilla.com/D140871
Depending on the subprocesses created (such as if
`--with-ccache=sccache` is set), `./mach configure` may hang.
More details about why this is is documented in bug 1753797.
The workaround is to lean on the standard library to stream-and-format
output, rather than our existing `ProcessHandler` code.
Though this still streams both `stdout` and `stderr` in real-time, I saw
some ordering differences between the two streams locally. I don't yet
have a reason to believe that these differences are harmful or otherwise
incorrect.
Differential Revision: https://phabricator.services.mozilla.com/D141028
On newer systems, Python is intelligently populating pure-python and
platform-specific packages into //two different directories//.
Traditionally, these directories would be symlinked - or only one would
exist. However, to support these newer systems (and be better integrated
with Python conventions), we should handle both of these directories
properly.
The solution here is, when calculating the `site_packages_dir()`,
calculate all of them, and handle the multitude at all call sites.
The one use case where we want only one path (the resolving the location
for the `mach.pth` pthfile), I've opted for the `platlib`, since both
pure python dependencies //and// platform-specific dependencies are
referenced by the pthfile.
Differential Revision: https://phabricator.services.mozilla.com/D140870
I'm guessing that the reason we haven't been building the Mach
virtualenv in CI so far is because:
* The default location in `~/.mozbuild` may not be cleared between jobs
* There's a performance cost to `pip install`-ing packages, especially
if they're not needed
* In the past, it was tricky to have some jobs use a Mach virtualenv and
others not.
This led to weird workarounds where a virtualenv would be created,
populated, then masqueraded as the "system Python" so that Mach would
be able to run with access to its dependencies without needing to
"create a Mach virtualenv".
This patch addresses the first point about the Mach virtualenv location:
In CI, if `WORKSPACE` is set, the Mach venv will be created in
`$WORKSPACE/mach_virtualenv`.
The other two points are solved by explicit configuration using the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
Differential Revision: https://phabricator.services.mozilla.com/D140256
There are cases in CI where we're using the system Python, but we
still want to create a virtualenv. For example, build jobs use the
system Python packages, but Mozharness wants to have access to Mach
packages. So, a virtualenv needs to be created (the `"common"`
virtualenv), whose associated `python` binary is used to invoke
Mozharness.
So, just like for the `build` site, allow creating the `common`
site even when the "native package source" is the system Python.
Since `mach_virtualenv_requirements.txt` already depends on
`common_virtualenv_requirements.txt`, this restriction should not cause
validation breakage.
Differential Revision: https://phabricator.services.mozilla.com/D140255
Adds support and documentation for the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
The "fallback" behaviour here should be backwards-compatible, which will
enable us to piece-by-piece explicitly use the `"system"` where
needed, then remove the backwards-compatibility shim afterwards.
A restriction added by this change is that it's no longer possible to
have the combination of:
* Have Mach use the system Python, and
* Have the active command site do `pip install`.
This is because this case shouldn't be necessary (if `pip install` is
allowed, why use the system?) and will allow us to enforce that all
non-build command sites always use their lockfiles, when they're set up.
When referring to behaviour in CI, I've opted to only refer to the
upcoming behaviour (of `MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"`)
because the current heuristic behaviour feels complex enough that it
should be understood by checking the code, rather than docs. Besides,
the heuristic will be removed pretty soon (right? ;)
Differential Revision: https://phabricator.services.mozilla.com/D138932
I'm guessing that the reason we haven't been building the Mach
virtualenv in CI so far is because:
* The default location in `~/.mozbuild` may not be cleared between jobs
* There's a performance cost to `pip install`-ing packages, especially
if they're not needed
* In the past, it was tricky to have some jobs use a Mach virtualenv and
others not.
This led to weird workarounds where a virtualenv would be created,
populated, then masqueraded as the "system Python" so that Mach would
be able to run with access to its dependencies without needing to
"create a Mach virtualenv".
This patch addresses the first point about the Mach virtualenv location:
In CI, if `WORKSPACE` is set, the Mach venv will be created in
`$WORKSPACE/mach_virtualenv`.
The other two points are solved by explicit configuration using the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
Differential Revision: https://phabricator.services.mozilla.com/D140256
There are cases in CI where we're using the system Python, but we
still want to create a virtualenv. For example, build jobs use the
system Python packages, but Mozharness wants to have access to Mach
packages. So, a virtualenv needs to be created (the `"common"`
virtualenv), whose associated `python` binary is used to invoke
Mozharness.
So, just like for the `build` site, allow creating the `common`
site even when the "native package source" is the system Python.
Since `mach_virtualenv_requirements.txt` already depends on
`common_virtualenv_requirements.txt`, this restriction should not cause
validation breakage.
Differential Revision: https://phabricator.services.mozilla.com/D140255
Adds support and documentation for the
`MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE` environment variable.
The "fallback" behaviour here should be backwards-compatible, which will
enable us to piece-by-piece explicitly use the `"system"` where
needed, then remove the backwards-compatibility shim afterwards.
A restriction added by this change is that it's no longer possible to
have the combination of:
* Have Mach use the system Python, and
* Have the active command site do `pip install`.
This is because this case shouldn't be necessary (if `pip install` is
allowed, why use the system?) and will allow us to enforce that all
non-build command sites always use their lockfiles, when they're set up.
When referring to behaviour in CI, I've opted to only refer to the
upcoming behaviour (of `MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE="none"`)
because the current heuristic behaviour feels complex enough that it
should be understood by checking the code, rather than docs. Besides,
the heuristic will be removed pretty soon (right? ;)
Differential Revision: https://phabricator.services.mozilla.com/D138932
We want `_pthfile_lines()` to consistently and accurately represent a
virtualenv's `sys.path` modifications.
However, this becomes tricky when comparing expected `pthfile_lines`
//before// activating a virtualenv versus //afterwards//, especially
since Python implicitly adds some paths (such as the path to
`site-packages`).
The current solution made this scrubbing only happen if we were
`pip install`-ing into the site. Unfortunately, it //doesn't// work for
the "use system Python for the `build` site", because:
* Pre-activation result: `site-packages` isn't added, because we aren't
using it
* Post-activation result, implicitly-added `site-packages` isn't
scrubbed, because **scrubbing only happened for
`SitePackagesSource.VENV`**
Stepping back, the solution here is:
* `pthfile_lines` only represents Mach modifications //on top// of
implicit virtualenv behaviour: since `site-packages` is always added
by default, it shouldn't be in our explicit `pthfile_lines`.
* The only time when implicit `sys.path` additions throws us off is when
`SitePackagesSource.SYSTEM`: so, move the scrubbing to happen in that
case instead.
Finally refactor the "conditional deprioritization" comment to be more
useful and more accurate: we've implemented the "nontrivial complexity"
of purging site-packages, and the other piece about nothing being
`pip install`-ed feels self-explanatory enough.
Differential Revision: https://phabricator.services.mozilla.com/D140580
So far, we've been using `virtualenv`'s `activate_this.py` script.
However, unlike earlier expectations, it adds its `sys.path` additions
to the //front//, not the back! This breaks our prioritization
requirements, such as:
* When using any package from the system environment, import *all
possible* packages from the system to avoid compatibility issues.
* Use vendored packages instead of virtualenv-installed packages
wherever possible, because it more-closely matches developer
expectations ("why is this package vendored if it's not used?")
Define an `activate_virtualenv()` function that replicates the logic
of `activate_this.py` [1], except for three differences:
* Don't modify `sys.real_prefix`, since it's a non-standard property of
`sys`.
* Only add seen-with-`venv`-module paths to the `sys.path` (`$prefix`,
`$prefix/.../$site_packages_dir`) - don't do the paths in-between.
* And, of course, append instead of prepend `sys.path` entries.
As an aside, this is one of the few remaining blockers from allowing
us to fully embrace `venv` instead of `virtualenv` - the last piece is
waiting on the fix for bug 1697833 to propagate.
[1]
https://github.com/pypa/virtualenv/blob/20.7.2/src/virtualenv/activation/python/activate_this.py
Differential Revision: https://phabricator.services.mozilla.com/D140579
Unless either the site's requirements or its on-disk virtualenv
changes, it shouldn't become out-of-date.
This will also catch situations where sites are incorrectly
being marked out-of-date due to `sys.path` mismanagement.
Differential Revision: https://phabricator.services.mozilla.com/D140578
When calling Mach, it was prefixed with the `srcdir`, even though the
`srcdir` was also the working directory of the invocation. This caused
the invocation to fail.
Differential Revision: https://phabricator.services.mozilla.com/D140400
Inconsistency confuses some of our tools. As part of this, I:
* Updated some docs to point to rust-minidump
* Added a fallback to mozcrash.py to try both versions
* Make mozcrash.py use --brief output when the local mdsw is used
* Remove the renaming hack from build-minidump-stackwalk.sh
This isn't as simple as a sed because we still have breakpad in tree
for minidump-analyzer. I did my best to replace the right strings.
Differential Revision: https://phabricator.services.mozilla.com/D138971
Updates parser documentation to clarify that, if no `paths` are
provided, then only those discovered by `--outgoing` and `--workdir`
are linted.
Differential Revision: https://phabricator.services.mozilla.com/D139158
Tie into `code_analysis`'s `get_clang_tools()` functionality to
intelligently bootstrap clang if it either doesn't exist or is
out-of-date.
This required exposing `command_context` to the linting logic, as it's
needed to call `artifact_toolchain(...)`.
Note that this means that the standalone `runcli.py` file won't be able
to support bootstrapping `clang-format`, or other linters that lean on
`command_context` in the future.
Finally, `substs.get("HOST_BIN_SUFFIX")` was replaced with a
windows-specific `binary += ".exe"`, because not all contexts where
the tests are run will have access to populated `substs` data.
Note that this worked before without the extension because it was
only used for starting a process, in which context Windows automatically
tries all `PATHEXT` options. Since we're now doing an `isfile()` check
(to enable more intelligent failure cases when `clang-format` doesn't
exist), we need the path to be fully correct.
Differential Revision: https://phabricator.services.mozilla.com/D137335
This unifies the ZIP permissions in omnijars and internal XPIs
produced by the packager directly and those repacked by l10n
single-repacks.
Differential Revision: https://phabricator.services.mozilla.com/D137340
Before, clang tools would only be bootstrapped if they didn't exist.
Now, bootstrapping also occurs if the version doesn't meet requirements.
Differential Revision: https://phabricator.services.mozilla.com/D137331
* Restructure "Using third-party Python packages" page to focus on the
"Mach commands"/"adding a Python package" use case since that's why
most people will be looking at these docs.
* Document the `<site>_virtualenv_packages.txt` behaviour and how it
relates to a Mach command's definition.
* Simplify the information around using a non-PyPI index to reference
the RelEng docs directly. It's a shame that the existing docs don't
explain how to identify tasks that need to use the internal mirror,
because I'm not sure either. There's existing cases of ad-hoc `pip`
installs //not// using the mirror, but the pattern isn't clear to me.
* Remove the "specify hashes" information, since the centralized
solution (will) automatically manage this internally.
* Arguably, it's still beneficial instructions for ad-hoc
`pip install` usages, but those are frowned upon today anyways - use
the centralized solution!
Differential Revision: https://phabricator.services.mozilla.com/D138931
This change allows IPDL sources to respect FINAL_LIBRARY when building, which
is important for allowing us to build gtest-only IPDL_SOURCES files.
Differential Revision: https://phabricator.services.mozilla.com/D137167
This patch adds support for generated files to be placed in
UNIFIED_SOURCES. This will be used in part 3 to build .cpp files
generated by IPDL_SOURCES by adding them to IPDL's UNIFIED_SOURCES under
the hood. Using a unified build for IPDL files is important, as a
significant build time regression was observed locally when using
non-unified sources to build ipdl sources, due to the large number of
generated files.
Differential Revision: https://phabricator.services.mozilla.com/D137166
This replaces all of the functions in gecko_taskgraph/util/taskcluster.py with
the ones from the vendored taskgraph if they are identical.
Differential Revision: https://phabricator.services.mozilla.com/D138458
andrej
Tom Ritter
ahochheiden
Alexandre Lissy
Dan Minor
Eemeli Aro
Mitchell Hentges
Cristian Tuns
Iulian Moraru
Alexis Beingessner
Nick Alexander
mvollmer
Nika Layzell
Andrew Halberstadt