The existing terminology had two issues:
* `VirtualenvManager` wasn't always associated with an on-disk
`virtualenv`: for example, when running in automation, Mach
"activates" a `VirtualenvManager`, updating its import scope,
but without ever creating an on-disk `virtualenv`.
* An upcoming patch splits the `VirtualenvManager` class, pulling
"on-disk virtualenv-handling functions" from the project-wide
interface for managing Python's import scope.
After some good discussion with Ahal, I think we've struck
the terminology that handles this distinction well: we'll call
the "import scope"-handling part the "site", and we'll continue
to call on-disk virtualenvs (and their representative classes)
as, well, virtualenvs.
Differential Revision: https://phabricator.services.mozilla.com/D130391
As of now, wine is used with:
- fxc, which is detected in toolkit/moz.configure
- midl, which is detected in toolkit/moz.configure
- ml64 or armasm64, for some assembly sources that aren't used in
projects that don't include toolkit/moz.configure.
We can thus move it there.
Differential Revision: https://phabricator.services.mozilla.com/D126166
As of now, wine is used with:
- fxc, which is detected in toolkit/moz.configure
- midl, which is detected in toolkit/moz.configure
- ml64 or armasm64, for some assembly sources that aren't used in
projects that don't include toolkit/moz.configure.
We can thus move it there.
As of bug 1731195, there is no remaining use of wine for artifact
builds, so we also make it limited to builds with a compile environment.
Differential Revision: https://phabricator.services.mozilla.com/D126166
Now that we bootstrap it, we don't need to look for old versioned
versions from mozilla-build. While here stop looking for makensis when
not building for Windows.
Differential Revision: https://phabricator.services.mozilla.com/D125906
By hybrid unified system we understand a system that encapsulates modules that are built in the unified mode
but also other modules, like `dom/Animation`, as an example, in the non unified environment.
This approach is desirable since we already have most of the modules transitioned to the non unified system but there are
still some that are not yet compatible, but in the long term this will be done by each module owner and can be also tested
locally using the build system.
If a module can't be built outside the unified method it's `moz.build` config file needs to have `REQUIRES_UNIFIED_BUILD = False`
To also enable this we need to have a flag from `mozconfig`, like:
```
ac_add_options --disable-unified-build
```
Differential Revision: https://phabricator.services.mozilla.com/D122328
By hybrid unified system we understand a system that encapsulates modules that are built in the unified mode
but also other modules, like `dom/Animation`, as an example, in the non unified environment.
This approach is desirable since we already have most of the modules transitioned to the non unified system but there are
still some that are not yet compatible, but in the long term this will be done by each module owner and can be also tested
locally using the build system.
If a module can't be built outside the unified method it's `moz.build` config file needs to have `REQUIRES_UNIFIED_BUILD = False`
To also enable this we need to have a flag from `mozconfig`, like:
```
ac_add_options --disable-unified-build
```
Differential Revision: https://phabricator.services.mozilla.com/D122328
Bug 1692137 added code that would end up calling
toolchain_task_definitions twice, once for each of host and target.
With the recent changes from bug 1719229, that is not useful anymore.
We also don't need the toolchain prefixes for toolchains on unsupported
hosts (which were useful before bug 1719229 for the target sysroots).
Differential Revision: https://phabricator.services.mozilla.com/D120044
In cross-compilation setups (x86_64 host, i686 or aarch64 target), we're
going to need two sysroots. Obviously, we need the sysroot paths to be
different in that case, so the sysroot path themselves need to contain
some distinctive name, and we'll use the `target.toolchain` name for
that (the target triplet with the vendor/machine stripped out).
Because the path name needs to be reflected in the artifact name as well
as the toolchain name, we also change them.
And because the current prefix in the toolchain name is now redundant
with the suffix, we remove the prefix, and allow the bootstrapping
mechanism to try toolchains without the prefix.
Differential Revision: https://phabricator.services.mozilla.com/D119846
As there are some things to solve first for the sysroot, we don't enable
the sysroot bootstrap unless --enable-bootstrap is given explicitly. The
default will still bootstrap everything else when building on central.
Differential Revision: https://phabricator.services.mozilla.com/D118450
This means --enable-bootstrap now is the same as what
--enable-bootstrap=install currently does. --enable-bootstrap=install is at the
same time desupported.
We also remove --enable-bootstrap=update, which is not worth supporting.
Differential Revision: https://phabricator.services.mozilla.com/D105719
This changes things such that setting e.g. NASM=/usr/bin/nasm will avoid
bootstrapping nasm even when bootstrapping is enabled.
This is not applied to CC/CXX/HOST_CC/HOST_CXX because things are more
complicated.
This also simplifies how check_prog is called for a bootstrapped tool,
and avoids the repetition of when.
CBINDGEN handling needs the pattern being applied manually because it
currently doesn't use check_prog. Once --enable-bootstrap=install
becomes the default on developer builds, it will be possible to simplify
this.
Differential Revision: https://phabricator.services.mozilla.com/D105718
Bug 1690930 added sysroots that can be bootstrapped. With this change,
we allow --enable-bootstrap=install to pull the right sysroot for the
configured target, and --enable-bootstrap to update it if it was already
there.
Differential Revision: https://phabricator.services.mozilla.com/D104797
We still default to updates only, though, because all the details of
only installing what's needed are not ironed out yet.
Differential Revision: https://phabricator.services.mozilla.com/D103865
This adds a --enable-bootstrap build flag that will automatically update
cbindgen, node, clang, sccache, nasm, wine, lucetc, dump_syms, pdbstr,
and winchecksec if they are already installed in ~/.mozbuild.
Eventually, we'll want to allow to install toolchains that weren't
already install, but one step at a time.
This explicitly doesn't cover rustc, which is its own can of worms, or
android-{ndk,sdk}, which are not installed via toolchain artifacts
currently.
Differential Revision: https://phabricator.services.mozilla.com/D101723
When MOZ_FETCHES_DIR is set, use that instead of ~/.mozbuild. This will
allow to find them on automation without explicit mozconfig help.
Differential Revision: https://phabricator.services.mozilla.com/D101719
Instead of adding all possible tool paths from ~/.mozbuild, we only
add the relevant paths for each of the tools we search for.
Differential Revision: https://phabricator.services.mozilla.com/D101718
Code review followups for 1675600: Restrict the use of new pass manager during LTO to builds where we're using the new pass manager in general, and (on Windows) where lld-link is new enough to understand the flag.
Differential Revision: https://phabricator.services.mozilla.com/D97372
Embedded symbols aren't useful for Android because:
* The associated Android tooling supports detached symbols
* The cost of the increased size is extra severe for mobile
However, Linux/Mac tooling isn't as supportive of detached symbols,
and the extra size isn't as impactful.
So, fully remove symbols for Android (even in profile builds), but keep
'em for other platforms.
There was a previous solution to remove profile-specific strip logic
from moz.configure and to instead adjust mozconfigs to achieve
the desired stripping amount. As requested in bug 1667170, this
implements the above^ solution after rolling back those previous
changesets:
* fd891240a96a
* 2f2a6df81762
* bbc6744b3ca2
* f849afa84d90
Differential Revision: https://phabricator.services.mozilla.com/D96902
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
`mach configure` currently runs the equivalent to `make -f client.mk`.
This is history, and essentially does the following:
- Create `configure` and `js/src/configure` from `configure.in` and
`js/src/configure.in` respectively.
- Create the objdir.
- Run `configure` from the objdir.
The `configure` script is, nowadays, only really used as a means to set
OLD_CONFIGURE (and also for people who want to run `configure`,
literally, as in the `configure; make` workflow). `mach configure`
actually doesn't need it. Neither does recursing into `js/src` require
`js/src/configure`, since bug 1520340 (and now as of bug 1669633, we
don't even recurse).
Because configure.py can actually derive OLD_CONFIGURE on its own
(except for `js/src/configure`, but `mach configure` doesn't run that),
we don't really need `configure` for `mach configure`.
So all in all, we're at a point in history where it's straightforward to
just initiate configure.py from mach configure, so we just do that.
And in the hypothetical case where the `mach configure` code is somehow
running in python2, we get the mach virtualenv python3 and use it to
execute `configure.py`.
Differential Revision: https://phabricator.services.mozilla.com/D93741
Rustc >= 1.44 changed the file names of the static libraries it
produces with -windows-gnu targets, to match that of mingw clang/gcc.
Considering we still build on 1.43, the best fix would be to derive the
prefix/suffix based on the version of rust, but that actually turns into
a hard-to-solve problem because of configure tests for bindgen also
depending on the prefix/suffix value to be known.
On the other hand, we're soon due to an update to 1.47, so the simpler
solution is to just push mingw builds to require 1.44 (settling for the
smallest upgrade possible for now) and to remove the split between C and
rust library prefix/suffixes.
Differential Revision: https://phabricator.services.mozilla.com/D93726
`mach configure` currently runs the equivalent to `make -f client.mk`.
This is history, and essentially does the following:
- Create `configure` and `js/src/configure` from `configure.in` and
`js/src/configure.in` respectively.
- Create the objdir.
- Run `configure` from the objdir.
The `configure` script is, nowadays, only really used as a means to set
OLD_CONFIGURE (and also for people who want to run `configure`,
literally, as in the `configure; make` workflow). `mach configure`
actually doesn't need it. Neither does recursing into `js/src` require
`js/src/configure`, since bug 1520340 (and now as of bug 1669633, we
don't even recurse).
Because configure.py can actually derive OLD_CONFIGURE on its own
(except for `js/src/configure`, but `mach configure` doesn't run that),
we don't really need `configure` for `mach configure`.
So all in all, we're at a point in history where it's straightforward to
just initiate configure.py from mach configure, so we just do that.
And in the hypothetical case where the `mach configure` code is somehow
running in python2, we get the mach virtualenv python3 and use it to
execute `configure.py`.
Differential Revision: https://phabricator.services.mozilla.com/D93741
Now that we don't recurse into the js python configure, we don't need to
have a special treatment for the options that need to be passed down to
that subconfigure, which is what js_option was for.
Differential Revision: https://phabricator.services.mozilla.com/D92727
In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically:
1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change.
2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781.
3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed.
As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages.
With that in mind, this patch does all the following:
1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it.
2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`.
3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`).
4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3).
5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes.
This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes:
1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI.
2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code.
Differential Revision: https://phabricator.services.mozilla.com/D85916
Upon reflection, we probably don't need the extra symbols anymore. Back
when we reduced the amount of stripping when enabling profiling, the
profiler wasn't able to download symbols on its own. It now is able to
do so, on all platforms.
As the stripping happens at packaging time, this doesn't change anything
for mach run on local builds.
Differential Revision: https://phabricator.services.mozilla.com/D86666
Versions of NSPR >= 4.10 come with a pkg-config file. We currently
depend on 4.9.2 for spidermonkey, but much more recent versions for
Firefox. 4.10 is less than a year newer than 4.9.2, and 4.10 is 7 years
old, so bumping the requirement to 4.10 is not really a big deal.
With the use of pkg-config, --with-nspr-cflags and --with-nspr-libs are
not needed.
None of the AC_TRY_COMPILE tests were any useful because
PR_STATIC_ASSERT and PR_UINT64 have been when we look for them since
4.8.6 and 4.9 respectively.
Differential Revision: https://phabricator.services.mozilla.com/D77412
As all versions of zlib >= 1.2.3.1 have a pkg-config file, and 1.2.3.1
is close to 14 years old, let's drop 1.2.3 and just use pkg-config, which
simplifies what we need to do dramatically.
Differential Revision: https://phabricator.services.mozilla.com/D77404
The need for --disable-install-strip in the mac mozconfigs comes from a
discrepancy in how stripping is handled between platforms. On Windows,
there is no stripping. On non-Mac unix, `strip` removes local symbols as
well as debug info and debug symbols. On Mac, it actually removes too
much, and one has to pass flags to remove both local symbols (`-x`) and
debug symbols (`-S`). Debug info is already in a separate file
(`.dSYM`).
For profiling reasons, we do ship e.g. nightlies with local symbols but
not debug info or symbols (or at least that's the intent). On Windows,
again, nothing to do. On non-Mac unix, we pass `--strip-debug` to
`strip` so that it keeps local symbols. That's where the discrepancy
comes in for Mac: the build system doesn't handle this at all, so the
mozconfigs contain --disable-install-strip to avoid stripping.
The build system should be doing what it's expected to be doing from the
start, without mozconfigs opting into anything.
AFAIK, we only really need the local symbols, so we can `strip -S` on
Mac when profiling is enabled, rather than `strip -x -S`. This also
significantly reduces the size of the installer for nightlies.
And while we're here, move the logic out of old-configure and into
python configure.
Differential Revision: https://phabricator.services.mozilla.com/D76789
This includes scripts that involve `tup`, jobs that build `tup` in automation, `tup.configure`, and related infrastructure and documentation.
Differential Revision: https://phabricator.services.mozilla.com/D73921
- Remove the separate option() for MT, because it dates back from when
we needed `MT` not being an absolute path, but that hasn't been true
since bug 1290040.
- Extend what was done in bug 1617794 to MT, although the long term move
is to not rely on MT at all.
- Patch leftovers from bug 1613799.
Differential Revision: https://phabricator.services.mozilla.com/D64712
--HG--
extra : moz-landing-system : lando
Windows programs run via Wine don't like Unix absolute paths (they look
like command line arguments), so we need to use relative paths.
Mingw already run fxc2 via wine, but for some reason it doesn't care
about the Unix absolute paths. genshaders does need some adjustements to
run properly with the real fxc.
Now, on actual Windows, because the temporary directory where
tempfile.NamedTemporaryFile creates files by default is not necessarily
on the same drive as where the command runs from, a relative path can't
be constructed. So we also force the temporary file to be created in the
current (obj) directory.
There is no similar concern for other files because we only go from
objdir to srcdir, and the build system already doesn't support both
being on a separate drive.
While here, flush stdout when the genshared script writes to it, so that
the messages are printed out immediately rather than randomly, later,
after output from subprocesses.
Differential Revision: https://phabricator.services.mozilla.com/D64294
--HG--
extra : moz-landing-system : lando
Mike Hommey
Ryan VanderMeulen
Mitchell Hentges
Greg Tatum
Marian-Vasile Laza
Emilio Cobos Álvarez
Cristian Tuns
Noemi Erli
Andi-Bogdan Postelnicu
Narcis Beleuzu
Butkovits Atila
Nick Alexander
Cosmin Sabou
David Major
Ricky Stewart
Bogdan Tara
Dorel Luca
Razvan Maries
Coroiu Cristina
Thinker Li
Andreea Pavel