libdav1d only has relative paths. Like ../../third_party/foo/bar.c
libjpeg-xl has absolute paths like /third_party/bar/foo.c
This handles the second case by not doing our relative path
correction logic if we don't need to.
It will find all of the types of ways a file is included in a
moz.build file* and iterate over them, trying the slightly
different logic. This is okay, because the failure scenario
for trying an incorrect mode is just that we don't get a match.
By trying all the modes we can increase our chances of finding a
match.
* For SOURCES/UNIFIED_SOURCES there aren't actually any situations
where we have mixed-modes in a moz.build file. But a future patch
will add support for editing EXPORTS and they do have mixed-modes.
Differential Revision: https://phabricator.services.mozilla.com/D120478
Bindgen will be needed for gecko_profiler::is_active gecko_profiler_label API
in this patch, as well as the marker API that will land after this.
Differential Revision: https://phabricator.services.mozilla.com/D120789
When transitioning from using `self` to using `command_context` in mach
commands, this specific mistake went undetected (no bustage forced a backout,
either). This commit fixes it.
Differential Revision: https://phabricator.services.mozilla.com/D120804
After removing `optional` in Bug 1712804, we need to add a variant back
here because there's fallible dependencies. However, I've tweaked the
re-introduction of the feature to require a specific repercussion
message as well. This seemed like a decent tradeoff - the developer
becomes aware that the failure is bad, it has repercussions, but it's
not a blocking issue. Additionally, since we're printing pip's output,
the developer will be able to see the underlying error causing the
warning.
I also added comment functionality to requirements definitions to allow
adjacent documentation of why some requirements are fallible. (Related:
I'm looking forward to `mach_bootstrap` not needing to parse
requirements definitions. Almost there!)
Note that we'll temporarily lose the "pinned" nature of the
three moved dependencies until dependency locking is implemented
for Mach requirements definitions. Also note that the pinned
`zstandard_requirements.txt` can't be removed like the other
files because it still has a dangling usage.
Finally, in preparation for review: I didn't make
`PypiOptionalSpecifier` extend `PypiSpecifier` because I figured that
the benefit of flexibility (easier to allow implementations to diverge
without needing to untangle an inheritance relationship) was larger than
the cost of needing to add properties to both specifiers.
If we wanted re-use, I'd probably have `PypiOptionalSpecifier` _contain_
a `PypiSpecifier`, but then you have to reach deeper into the object to
get data, so *shrug*.
Differential Revision: https://phabricator.services.mozilla.com/D119835
`_install_pip_package()` may be run from `populate()`, which
is invoked from a child Python process that doesn't
have in-tree Python modules in its sys.path.
An alternate solution would be to add in-tree modules
to the sys.path, but that seemed more costly than
simply using `tempfile` and `shutil`.
Differential Revision: https://phabricator.services.mozilla.com/D119834
After removing `optional` in Bug 1712804, we need to add a variant back
here because there's fallible dependencies. However, I've tweaked the
re-introduction of the feature to require a specific repercussion
message as well. This seemed like a decent tradeoff - the developer
becomes aware that the failure is bad, it has repercussions, but it's
not a blocking issue. Additionally, since we're printing pip's output,
the developer will be able to see the underlying error causing the
warning.
I also added comment functionality to requirements definitions to allow
adjacent documentation of why some requirements are fallible. (Related:
I'm looking forward to `mach_bootstrap` not needing to parse
requirements definitions. Almost there!)
Note that we'll temporarily lose the "pinned" nature of the
three moved dependencies until dependency locking is implemented
for Mach requirements definitions. Also note that the pinned
`zstandard_requirements.txt` can't be removed like the other
files because it still has a dangling usage.
Finally, in preparation for review: I didn't make
`PypiOptionalSpecifier` extend `PypiSpecifier` because I figured that
the benefit of flexibility (easier to allow implementations to diverge
without needing to untangle an inheritance relationship) was larger than
the cost of needing to add properties to both specifiers.
If we wanted re-use, I'd probably have `PypiOptionalSpecifier` _contain_
a `PypiSpecifier`, but then you have to reach deeper into the object to
get data, so *shrug*.
Differential Revision: https://phabricator.services.mozilla.com/D119835
`_install_pip_package()` may be run from `populate()`, which
is invoked from a child Python process that doesn't
have in-tree Python modules in its sys.path.
An alternate solution would be to add in-tree modules
to the sys.path, but that seemed more costly than
simply using `tempfile` and `shutil`.
Differential Revision: https://phabricator.services.mozilla.com/D119834
Now that the upstream Python bug has been resolved since Python 3.4 (at
the latest), we can safely remove the environment variable workaround.
Differential Revision: https://phabricator.services.mozilla.com/D119687
It's possible for a virtualenv to have an incorrect list of directories
to add to the sys.path, such as the following cases:
* Its creation got cancelled halfway through
* The list of pths changed in a new revision
* It got modified by an external tool
By validating the list of provided pths against the list of required
pths, we can ensure that the virtualenv is more dependably up-to-date.
Differential Revision: https://phabricator.services.mozilla.com/D119686
This simplifies consumer logic, since they get the parsed list of pypi
and pth requirements, as well as the list of input files that were
parsed.
One benefit of this simplification is that we no longer
recursively create VirtualenvManagers.
Note that mach_bootstrap cannot (yet) take advantage
of `ParseMachEnvRequirements` because of a dependency cycle:
* `mach_bootstrap` must set up the `sys.path` to import
`ParseMachEnvRequirements`.
* `mach_bootstrap` would want `ParseMachEnvRequirements` to
determine which paths to add to the `sys.path`.
Differential Revision: https://phabricator.services.mozilla.com/D119685
ensure_browser_packages is used for another purpose since Bug 1718341.
this also fixed a missing overload for ensure_android_artifact_mode_packages
and ensure_js_packages and some missing arguments in install_android_packages.
Differential Revision: https://phabricator.services.mozilla.com/D120511
After removing `optional` in Bug 1712804, we need to add a variant back
here because there's fallible dependencies. However, I've tweaked the
re-introduction of the feature to require a specific repercussion
message as well. This seemed like a decent tradeoff - the developer
becomes aware that the failure is bad, it has repercussions, but it's
not a blocking issue. Additionally, since we're printing pip's output,
the developer will be able to see the underlying error causing the
warning.
I also added comment functionality to requirements definitions to allow
adjacent documentation of why some requirements are fallible. (Related:
I'm looking forward to `mach_bootstrap` not needing to parse
requirements definitions. Almost there!)
Note that we'll temporarily lose the "pinned" nature of the
three moved dependencies until dependency locking is implemented
for Mach requirements definitions. Also note that the pinned
`zstandard_requirements.txt` can't be removed like the other
files because it still has a dangling usage.
Finally, in preparation for review: I didn't make
`PypiOptionalSpecifier` extend `PypiSpecifier` because I figured that
the benefit of flexibility (easier to allow implementations to diverge
without needing to untangle an inheritance relationship) was larger than
the cost of needing to add properties to both specifiers.
If we wanted re-use, I'd probably have `PypiOptionalSpecifier` _contain_
a `PypiSpecifier`, but then you have to reach deeper into the object to
get data, so *shrug*.
Differential Revision: https://phabricator.services.mozilla.com/D119835
`_install_pip_package()` may be run from `populate()`, which
is invoked from a child Python process that doesn't
have in-tree Python modules in its sys.path.
An alternate solution would be to add in-tree modules
to the sys.path, but that seemed more costly than
simply using `tempfile` and `shutil`.
Differential Revision: https://phabricator.services.mozilla.com/D119834
Now that the upstream Python bug has been resolved since Python 3.4 (at
the latest), we can safely remove the environment variable workaround.
Differential Revision: https://phabricator.services.mozilla.com/D119687
It's possible for a virtualenv to have an incorrect list of directories
to add to the sys.path, such as the following cases:
* Its creation got cancelled halfway through
* The list of pths changed in a new revision
* It got modified by an external tool
By validating the list of provided pths against the list of required
pths, we can ensure that the virtualenv is more dependably up-to-date.
Differential Revision: https://phabricator.services.mozilla.com/D119686
This simplifies consumer logic, since they get the parsed list of pypi
and pth requirements, as well as the list of input files that were
parsed.
One benefit of this simplification is that we no longer
recursively create VirtualenvManagers.
Note that mach_bootstrap cannot (yet) take advantage
of `ParseMachEnvRequirements` because of a dependency cycle:
* `mach_bootstrap` must set up the `sys.path` to import
`ParseMachEnvRequirements`.
* `mach_bootstrap` would want `ParseMachEnvRequirements` to
determine which paths to add to the `sys.path`.
Differential Revision: https://phabricator.services.mozilla.com/D119685
The very first time an Android AVD starts it runs some one time jobs to
properly set up the AVD.
To avoid running the setup every time we run tests in automation, we can boot
the AVD before packaging it so that the testing jobs can use a "prewarmed" AVD
instead.
Differential Revision: https://phabricator.services.mozilla.com/D119225
Tooltool images are hard to update because we don't provide a script to
generate the image and documentation is often inaccurate.
This patch makes it so we generate the AVD in the android-sdk TL job instead.
Differential Revision: https://phabricator.services.mozilla.com/D119221
After removing `optional` in Bug 1712804, we need to add a variant back
here because there's fallible dependencies. However, I've tweaked the
re-introduction of the feature to require a specific repercussion
message as well. This seemed like a decent tradeoff - the developer
becomes aware that the failure is bad, it has repercussions, but it's
not a blocking issue. Additionally, since we're printing pip's output,
the developer will be able to see the underlying error causing the
warning.
I also added comment functionality to requirements definitions to allow
adjacent documentation of why some requirements are fallible. (Related:
I'm looking forward to `mach_bootstrap` not needing to parse
requirements definitions. Almost there!)
Note that we'll temporarily lose the "pinned" nature of the
three moved dependencies until dependency locking is implemented
for Mach requirements definitions. Also note that the pinned
`zstandard_requirements.txt` can't be removed like the other
files because it still has a dangling usage.
Finally, in preparation for review: I didn't make
`PypiOptionalSpecifier` extend `PypiSpecifier` because I figured that
the benefit of flexibility (easier to allow implementations to diverge
without needing to untangle an inheritance relationship) was larger than
the cost of needing to add properties to both specifiers.
If we wanted re-use, I'd probably have `PypiOptionalSpecifier` _contain_
a `PypiSpecifier`, but then you have to reach deeper into the object to
get data, so *shrug*.
Differential Revision: https://phabricator.services.mozilla.com/D119835
`_install_pip_package()` may be run from `populate()`, which
is invoked from a child Python process that doesn't
have in-tree Python modules in its sys.path.
An alternate solution would be to add in-tree modules
to the sys.path, but that seemed more costly than
simply using `tempfile` and `shutil`.
Differential Revision: https://phabricator.services.mozilla.com/D119834
Now that the upstream Python bug has been resolved since Python 3.4 (at
the latest), we can safely remove the environment variable workaround.
Differential Revision: https://phabricator.services.mozilla.com/D119687
It's possible for a virtualenv to have an incorrect list of directories
to add to the sys.path, such as the following cases:
* Its creation got cancelled halfway through
* The list of pths changed in a new revision
* It got modified by an external tool
By validating the list of provided pths against the list of required
pths, we can ensure that the virtualenv is more dependably up-to-date.
Differential Revision: https://phabricator.services.mozilla.com/D119686
This simplifies consumer logic, since they get the parsed list of pypi
and pth requirements, as well as the list of input files that were
parsed.
One benefit of this simplification is that we no longer
recursively create VirtualenvManagers.
Note that mach_bootstrap cannot (yet) take advantage
of `ParseMachEnvRequirements` because of a dependency cycle:
* `mach_bootstrap` must set up the `sys.path` to import
`ParseMachEnvRequirements`.
* `mach_bootstrap` would want `ParseMachEnvRequirements` to
determine which paths to add to the `sys.path`.
Differential Revision: https://phabricator.services.mozilla.com/D119685
The very first time an Android AVD starts it runs some one time jobs to
properly set up the AVD.
To avoid running the setup every time we run tests in automation, we can boot
the AVD before packaging it so that the testing jobs can use a "prewarmed" AVD
instead.
Differential Revision: https://phabricator.services.mozilla.com/D119225
Tooltool images are hard to update because we don't provide a script to
generate the image and documentation is often inaccurate.
This patch makes it so we generate the AVD in the android-sdk TL job instead.
Differential Revision: https://phabricator.services.mozilla.com/D119221
This step removes all the dependencies of mach commands to
having a MachCommandBase as the `self` by using the `command_context`
argument instead. This also removes any remaining statefulness from those
classes that implement mach commands, ultimately making it easier to move
existing commands out of classes in a follow-up.
Differential Revision: https://phabricator.services.mozilla.com/D118058
We will only run the processes in CET compatible modules only mode when not
using the JIT code. So marking xul.dll as compatible should be OK.
Differential Revision: https://phabricator.services.mozilla.com/D117551
I'm guessing that the code moved to the bootstrap command but the
associated comment was forgotten and left abandoned, which is very sad.
Differential Revision: https://phabricator.services.mozilla.com/D120098
There's a "minimum Python check" that happens when Mach
bootstraps that makes the bootstrap command's check obsolete.
Since we want to use modern Python features throughout Mach, we need to
move Python-out-of-date logic up to the front of the process.
We're duplicating the OS-detection logic a bit, but now we regain
per-platform Python instructions.
Differential Revision: https://phabricator.services.mozilla.com/D120093
We no longer require (or even allow) Python 2 for Mach, so the
"missing Python 2" warning is obsolete and just causes confusion.
Differential Revision: https://phabricator.services.mozilla.com/D120092
The very first time an Android AVD starts it runs some one time jobs to
properly set up the AVD.
To avoid running the setup every time we run tests in automation, we can boot
the AVD before packaging it so that the testing jobs can use a "prewarmed" AVD
instead.
Differential Revision: https://phabricator.services.mozilla.com/D119225
Tooltool images are hard to update because we don't provide a script to
generate the image and documentation is often inaccurate.
This patch makes it so we generate the AVD in the android-sdk TL job instead.
Differential Revision: https://phabricator.services.mozilla.com/D119221
This step removes all the dependencies of mach commands to
having a MachCommandBase as the `self` by using the `command_context`
argument instead. This also removes any remaining statefulness from those
classes that implement mach commands, ultimately making it easier to move
existing commands out of classes in a follow-up.
Differential Revision: https://phabricator.services.mozilla.com/D118058
At the moment, lucetc is not useful by default, only when enabling wasm
sandboxed libraries manually, and in that case, configure knows to
bootstrap lucetc itself. Furthermore, lucetc is set to be replaced with
wasm2c, which is in-tree and doesn't require bootstrapping at all.
Differential Revision: https://phabricator.services.mozilla.com/D120051
The very first time an Android AVD starts it runs some one time jobs to
properly set up the AVD.
To avoid running the setup every time we run tests in automation, we can boot
the AVD before packaging it so that the testing jobs can use a "prewarmed" AVD
instead.
Differential Revision: https://phabricator.services.mozilla.com/D119225
Tooltool images are hard to update because we don't provide a script to
generate the image and documentation is often inaccurate.
This patch makes it so we generate the AVD in the android-sdk TL job instead.
Differential Revision: https://phabricator.services.mozilla.com/D119221
Bug 1719229 broke the install by accident, but configure takes care of
it by default since bug 1716911, bug 1716912 or bug 1717585 depending on
the platform, so we can just stop installing it from mach bootstrap.
Differential Revision: https://phabricator.services.mozilla.com/D119933
Some developers hold onto a `comm` directory as a mount point
that they conditionally populate depending on whether they're working on
Firefox or Thunderbird.
The "`comm` directory exists == is a thunderbird checkout"
assumption isn't compatible with this workflow.
The fix embraced by this patch is to check if the `comm` directory has
any contents.
Differential Revision: https://phabricator.services.mozilla.com/D119193
These are enough for me to run bootstrap+configure+build.
Some touch third-party code (gyp), but per discussion in the earlier
versions of this revision that seems fine.
Differential Revision: https://phabricator.services.mozilla.com/D119080
When running `mach python-test` on a clobbered tree, we create both the
python-test and common virtualenvs, but the former should be enough. The
latter is only created when running tests that use the BaseConfigureTest
class, when it indirectly includes init.configure, which makes the
configure sandbox run the virtualenv_python3 function, which ends up
initializing the common virtualenv. We only don't end up re-executing
that virtualenv python thanks to 6680ca0acc27 which was a workaround
that would allow any virtualenv in $objdir/_virtualenvs, while still
creating a virtualenv it won't use.
Tests now tell virtualenv_python3 to not use the normal virtualenv
manager, and remove the workaround.
Differential Revision: https://phabricator.services.mozilla.com/D118775
These are enough for me to run bootstrap+configure+build.
Some touch third-party code (gyp), but per discussion in the earlier
versions of this revision that seems fine.
Differential Revision: https://phabricator.services.mozilla.com/D119080
When we customized the list of possible `applications` for M1 Macs, the
change was only applying to the displayed list, but not the actual
list that the choice was indexed against.
This is fixed by properly using `applications` instead of
`APPLICATIONS` where relevant.
Note that it's still possible to manually request an
`artifact_mode` if you provide it as an argument for `./mach bootstrap`,
but this workflow is usually only done by devs who are familiar with all
the applications, and therefore likely know what they're doing.
Differential Revision: https://phabricator.services.mozilla.com/D118625
This was added ing bug 1352595. It has only been used briefly, and we
since then have gone all the way to never compress omni.ja. Even if we
did go back to compressing, we'd probably go with zstd rather than
brotli.
The gecko-side support for this has only ever been nightly-only.
Differential Revision: https://phabricator.services.mozilla.com/D118655
`_get_infer_config()` returns 3 items, but the existing code expects 2.
Add a third `_` to capture the other unwanted item.
Differential Revision: https://phabricator.services.mozilla.com/D118535
wireless-tools-devel isn't available on Rocky Linux. Fortunately, it doesn't appear to be needed for the build anymore, and it was added almost a decade ago, so it should be safe to remove.
Differential Revision: https://phabricator.services.mozilla.com/D118532
In the existing behaviour, we would return a node/npm executable from
`find_executable()`, even if we couldn't extract a version from it or if
the extracted version was insufficient.
Now, we fail early if the detected node/npm is insufficient.
Differential Revision: https://phabricator.services.mozilla.com/D118377
As :julienw beautifully described on Bugzilla,
we no longer need to search for "nodejs" since all
packages providing Node >= v12 will provide the `node`
binary.
See his comment here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1716600#c4
Base-toolchain tasks with Node v10 should still be
happy because, unlike Debian's Node v10, they provide
a `node` binary instead of `nodejs`.
Differential Revision: https://phabricator.services.mozilla.com/D118375
(Tl;dr: The original author of this patch was axel@mozilla.com (:pike).
This patch has been modified to run in production automation.)
Cross-channel takes all the en-US strings from the shipping train branches
and consolidates them into a single repository. Originally, this ran out of
https://hg.mozilla.org/users/axel_mozilla.com/cross-channel-experimental/ .
The original version was coupled tightly with specific hg internals, making
hg upgrades fragile. Axel wrote
https://bug1659691.bmoattachments.org/attachment.cgi?id=9170636 before
handing off project ownership; this is the original patch to stop replaying
history to simplify the logic.
This patch also automates the previously manual preparation steps, allows
for running both in automation and locally, and adds retries for production
robustness.
Differential Revision: https://phabricator.services.mozilla.com/D116537
Having separate `<name>.pth` files in the virtual environments
isn't providing an advantage. We can simplify configuration
by putting all `pth` adjustments into a single file: `mach.pth`.
Differential Revision: https://phabricator.services.mozilla.com/D117710
This has two benefits:
1. `handle_package()` becomes more clear - rather than referring to
`action` and `package` with array index numbers, we now give
them real names. The benefit here is also shown in `up_to_date()`.
2. This makes the top-level parser for `packages()` less opinionated
about sub-formats: if an action has a nested structure, it should
have the flexibility to define what it looks like.
Differential Revision: https://phabricator.services.mozilla.com/D117708
After some testing in `try` and locally, the manual
`PYTHONEXECUTABLE` definitions shouldn't be needed
anymore.
There's been some work on Brew's
Python to improve its `sitecustomize` behaviour.
The most likely improvement is:
https://github.com/Homebrew/homebrew-core/pull/65297
However, I'm not fully confident in this change.
If it fails, it's more likely to affect developers
than CI. I think it's worth attempting a landing,
because if the variable is indeed obsolete, then
deleting it will avoid some spicy "action-at-a-
distance" behaviour.
Differential Revision: https://phabricator.services.mozilla.com/D117452
Vendoring wheels has three benefits:
* There's far less files, so Firefox checkouts will be smaller.
* It works around `zipp` not allowing `pip install`
from extracted source `tar.gz` files. Now, we should
be able to use the pip resolver against vendored
packages, which will be needed for future
mach virtualenv work.
* `./mach vendor python` takes far less time to execute.
Since we need the raw Python to be available to add to the `sys.path`,
we extract the wheels before putting them in tree.
Due to the structure of some wheels being less nested
than of a source `tar.gz`, `common_virtualenv_packages`
needed to be adjusted accordingly.
`install_pip_package()` had to be tweaked as well since you can't
`pip install` an extracted wheel. So, we "re-bundle" the wheel
before installing from a vendored package.
Replace python packages with wheels where possible
This contains the vendoring changes caused by the
last patch.
For reviewing, there's a couple things to note:
* A bunch of files are deleted, since there's generally
less files in a wheel than in a source archive.
* There's a new `.dist-info` directory for each
extracted wheel, so expect roughly 5 or
6 new files for each wheel'd package.
* There should be no source code changes other than
moves from package names changing from having
`-` to having `_`.
Differential Revision: https://phabricator.services.mozilla.com/D116512
As pointed out by Harry, this will help in the short-term with
onboarding new developers who will be using M1 macs.
Note that this patch doesn't leverage the "instance" classes to define
whether they're artifact-mode compatible or not, and that's because:
* Most (all except one?) of the systems support artifact mode, therefore
* Since this is a temporary workaround, it's more deletable to have
a top-level `if/else` than to add a `supports_artifact_mode` property
to each Bootstrapper.
Differential Revision: https://phabricator.services.mozilla.com/D117946
Last year, we stopped vendoring Python packages that have native
code. Since we have only had pure-python packages since, the
Windows-specific qualifier (or excluder in the case of `!windows`)
hasn't been needed.
I don't foresee us needing it again, but if anything we can peel it
back from `hg` history if this assumption is incorrect.
Differential Revision: https://phabricator.services.mozilla.com/D117468
On Windows, Python's temporary directories are placed in `%APPDATA%`.
Unfortunately, the failing tests want to assume that some "other"
path (as a subpath of a tempdir) won't be within any
`filter_args()` known top-level paths, which includes the user's home
dir.
The only reason that this assumption has passed in CI so far is due
to Windows case insensitivity: running locally had `~` evaluated
to `c:\Users\...`, while running in CI had it be `c:\users\...`.
Since the tempdir path started with `C:\Users`, `filter_args()`
would `<path omitted>` in CI but would `$HOME/...` locally.
By not basing the `other_path` on `tempdir`, we now consistently get the
same results across platforms.
Differential Revision: https://phabricator.services.mozilla.com/D117698
The `calls` parameter is expected to be an iterable
container of calls, not a singular call.
This was working in `mock-1.0.0` because `calls`
was (confusingly) allowed to be a single item
if `any_order==False`. This behaviour isn't
the same in the standard library.
Differential Revision: https://phabricator.services.mozilla.com/D117074
`./mach vendor python <package>` was already adding its new package
to `requirements.txt`, so we were getting full resolver support, which
is good. However, it caused our `requirements.in` file to start
getting out-of-date, and therefore made it harder to identify the
top-level dependencies.
Arguably, we could have `./mach vendor python <package>` automatically
update `requirements.in`, too, but then we need to solve the edge cases,
such as "What if the package is already in `requirements.in`? What if
that existing item has a different version?"
The hardest part of updating `requirements.in` was finding it, so I've
also modified the `./mach vendor python` help text to make it more
identifiable.
Differential Revision: https://phabricator.services.mozilla.com/D116386
For "c" (i.e., gcc, clang, and clang-cl), this configures the
`-f{debug,macro}-path-prefix` flags. We'd prefer to use
`-ffile-path-prefix`, but it seems that `clang-cl` does not recognize
that flag.
For "rust" (i.e., rustc/cargo), this configures `--remap-path-prefix`.
This is one step toward getting `sccache` hits across source and
object directories.
Differential Revision: https://phabricator.services.mozilla.com/D113065