In cross-compilation setups (x86_64 host, i686 or aarch64 target), we're
going to need two sysroots. Obviously, we need the sysroot paths to be
different in that case, so the sysroot path themselves need to contain
some distinctive name, and we'll use the `target.toolchain` name for
that (the target triplet with the vendor/machine stripped out).
Because the path name needs to be reflected in the artifact name as well
as the toolchain name, we also change them.
And because the current prefix in the toolchain name is now redundant
with the suffix, we remove the prefix, and allow the bootstrapping
mechanism to try toolchains without the prefix.
Differential Revision: https://phabricator.services.mozilla.com/D119846
It was enabled in bug 1712633 by mistake.
Because there wasn't a way to distinguish esr yet, we add one, and while
at it, expose it to the build system.
Differential Revision: https://phabricator.services.mozilla.com/D119682
As written in 00c93ac66282, relying on the 2.27-3ubuntu1.2 package being
available is a timebomb. We can however get the older sources from
launchpad, and build them ourselves.
Differential Revision: https://phabricator.services.mozilla.com/D119258
This allows to use the same toolchain docker images as other toolchains,
based on Debian buster.
While here, use the default max-run-time, which is more than enough for
this toolchain.
Differential Revision: https://phabricator.services.mozilla.com/D119137
Because GCC is built in stages, the final stage is built with
intermediate stages's GCC, which handles the sysroot correctly, so we
end up with headers and libraries with the expected compatibility.
This allows to use the same toolchain docker images as other toolchains,
based on Debian buster.
Differential Revision: https://phabricator.services.mozilla.com/D119136
- we needed -gcc-toolchain to pick C/C++ standard headers from the right
version of GCC, but we now have them in the toolchain sysroot (bug
1719207), so we can use that instead.
- we needed LD_LIBRARY_PATH when clang was built on an older version of
Debian, but that was changed in bug 1694775.
Differential Revision: https://phabricator.services.mozilla.com/D119135
We have a separate binutils toolchain already, and the only remaining
use of the binutils part of the the GCC toolchain is for the gold plugin
headers for clang, which we can add to the toolchain sysroot.
Differential Revision: https://phabricator.services.mozilla.com/D119133
When running `mach python-test` on a clobbered tree, we create both the
python-test and common virtualenvs, but the former should be enough. The
latter is only created when running tests that use the BaseConfigureTest
class, when it indirectly includes init.configure, which makes the
configure sandbox run the virtualenv_python3 function, which ends up
initializing the common virtualenv. We only don't end up re-executing
that virtualenv python thanks to 6680ca0acc27 which was a workaround
that would allow any virtualenv in $objdir/_virtualenvs, while still
creating a virtualenv it won't use.
Tests now tell virtualenv_python3 to not use the normal virtualenv
manager, and remove the workaround.
Differential Revision: https://phabricator.services.mozilla.com/D118775
I removed the -Wno-range-loop-analysis warning flag in bug 1718408 because I didn't see any build failures on Try without it, but apparently I missed some.
Differential Revision: https://phabricator.services.mozilla.com/D119039
This requires a backport of valgrind to buster instead of jessie (which
we don't need a backport for anymore). Somehow the buster-based
android-build was using the backport for jessie. It now can use the
backport for buster.
We now also need a few extra packages in the valgrind docker image that
used to be installed as a side effect of installing other packages, but
aren't installed automatically anymore, while necessary.
This allows to remove the debian8-amd64-build docker image, now unused.
We add a patch to workaround an issue with rust that was fixed in older
version of valgrind but that resurfaced in a slightly different manner.
Filed upstream as https://bugs.kde.org/show_bug.cgi?id=433641.
Finally, we update the suppressions to account for system changes.
Differential Revision: https://phabricator.services.mozilla.com/D106402
This requires a backport of valgrind to buster instead of jessie (which
we don't need a backport for anymore). Somehow the buster-based
android-build was using the backport for jessie. It now can use the
backport for buster.
We now also need a few extra packages in the valgrind docker image that
used to be installed as a side effect of installing other packages, but
aren't installed automatically anymore, while necessary.
This allows to remove the debian8-amd64-build docker image, now unused.
We add a patch to workaround an issue with rust that was fixed in older
version of valgrind but that resurfaced in a slightly different manner.
Filed upstream as https://bugs.kde.org/show_bug.cgi?id=433641.
Finally, we update the suppressions to account for system changes.
Differential Revision: https://phabricator.services.mozilla.com/D106402
As there are some things to solve first for the sysroot, we don't enable
the sysroot bootstrap unless --enable-bootstrap is given explicitly. The
default will still bootstrap everything else when building on central.
Differential Revision: https://phabricator.services.mozilla.com/D118450
The `mozbase` modules were being unconditionally added to the
`sys.path` regardless of the Mach command being run, so there isn't
much value keeping them in a separate file. Besides, all other
source module paths are described in `common_virtualenv_packages`,
why is `mozbase` special?
In the future, we're going to want to make improvements here (such as:
there's a difference between informing mach of first-party code
versus defining which third_party vendored packages should be in scope,
and that workflow difference should be represented in-code).
It's useful to peel out the existing, less useful abstraction before
we can build a stronger one.
Differential Revision: https://phabricator.services.mozilla.com/D117711
Having separate `<name>.pth` files in the virtual environments
isn't providing an advantage. We can simplify configuration
by putting all `pth` adjustments into a single file: `mach.pth`.
Differential Revision: https://phabricator.services.mozilla.com/D117710
This has two benefits:
1. `handle_package()` becomes more clear - rather than referring to
`action` and `package` with array index numbers, we now give
them real names. The benefit here is also shown in `up_to_date()`.
2. This makes the top-level parser for `packages()` less opinionated
about sub-formats: if an action has a nested structure, it should
have the flexibility to define what it looks like.
Differential Revision: https://phabricator.services.mozilla.com/D117708
After some testing in `try` and locally, the manual
`PYTHONEXECUTABLE` definitions shouldn't be needed
anymore.
There's been some work on Brew's
Python to improve its `sitecustomize` behaviour.
The most likely improvement is:
https://github.com/Homebrew/homebrew-core/pull/65297
However, I'm not fully confident in this change.
If it fails, it's more likely to affect developers
than CI. I think it's worth attempting a landing,
because if the variable is indeed obsolete, then
deleting it will avoid some spicy "action-at-a-
distance" behaviour.
Differential Revision: https://phabricator.services.mozilla.com/D117452
Vendoring wheels has three benefits:
* There's far less files, so Firefox checkouts will be smaller.
* It works around `zipp` not allowing `pip install`
from extracted source `tar.gz` files. Now, we should
be able to use the pip resolver against vendored
packages, which will be needed for future
mach virtualenv work.
* `./mach vendor python` takes far less time to execute.
Since we need the raw Python to be available to add to the `sys.path`,
we extract the wheels before putting them in tree.
Due to the structure of some wheels being less nested
than of a source `tar.gz`, `common_virtualenv_packages`
needed to be adjusted accordingly.
`install_pip_package()` had to be tweaked as well since you can't
`pip install` an extracted wheel. So, we "re-bundle" the wheel
before installing from a vendored package.
Replace python packages with wheels where possible
This contains the vendoring changes caused by the
last patch.
For reviewing, there's a couple things to note:
* A bunch of files are deleted, since there's generally
less files in a wheel than in a source archive.
* There's a new `.dist-info` directory for each
extracted wheel, so expect roughly 5 or
6 new files for each wheel'd package.
* There should be no source code changes other than
moves from package names changing from having
`-` to having `_`.
Differential Revision: https://phabricator.services.mozilla.com/D116512
Last year, we stopped vendoring Python packages that have native
code. Since we have only had pure-python packages since, the
Windows-specific qualifier (or excluder in the case of `!windows`)
hasn't been needed.
I don't foresee us needing it again, but if anything we can peel it
back from `hg` history if this assumption is incorrect.
Differential Revision: https://phabricator.services.mozilla.com/D117468
The `mach_bootstrap:search_path()` implementation is out of
date compared to `virtualenv.py`.
Since `python2:`, `python3:` and `optional:` packages are no
longer valid virtualenv requirement actions, they can be removed.
Differential Revision: https://phabricator.services.mozilla.com/D117707
For "c" (i.e., gcc, clang, and clang-cl), this configures the
`-f{debug,macro}-path-prefix` flags. We'd prefer to use
`-ffile-path-prefix`, but it seems that `clang-cl` does not recognize
that flag.
For "rust" (i.e., rustc/cargo), this configures `--remap-path-prefix`.
This is one step toward getting `sccache` hits across source and
object directories.
Differential Revision: https://phabricator.services.mozilla.com/D113065
Based on the docs and the code within `pathlib2`, it seems to focus
only on backporting the main features of `pathlib` to be available for
Python 2. It does _not_ provide features in newer Python versions
of `pathlib` (such as `is_mount`, new in Python 3.7). Therefore, it
doesn't provide anything that the standard library of Python 3.6+
doesn't have.
Differential Revision: https://phabricator.services.mozilla.com/D117071
Mike Hommey
lyavor
Chris Peterson
Alexandru Michis
Benjamin Beurdouche
Kai Engert
Mitchell Hentges
Alexis Beingessner
surajeet310
Nick Alexander
Julien Cristau
Deian Stefan
Andrew Osmond