This is not something that happens under normal circumstances, but it
can happen when you go fancy and run multiple configures in parallel
with different objdirs, and old-configure doesn't exist in the first
place ; then one configure may overwrite old-configure while another is
starting to execute it, resulting in the latter nor executing
old-configure completely.
Differential Revision: https://phabricator.services.mozilla.com/D42428
--HG--
extra : moz-landing-system : lando
Just like C++14 sized deallocation support, we don't want to support
this. We shouldn't be using `new` on over-aligned types anyway.
Differential Revision: https://phabricator.services.mozilla.com/D41819
--HG--
extra : moz-landing-system : lando
Some recent changes to how we set cross-language LTO for Windows
resulted in compilation-time decreases and small performance regressions
on a few benchmarks. The changes intended to remove explicit enablement
of cross-language LTO for all builds, but rely on shippable builds being
built with PGO and moz.configure's clever defaulting of cross-language
LTO for PGO'd builds on Windows, which would then enable cross-language
LTO for only shippable builds.
Obviously something went wrong with those changes.
The problem was our defaulting wasn't visible to moz.configure's logic
for how to pass command-line options to the JS subconfigure. We set the
value (`cross`) after the value for `--enable-lto` has been determined,
and the default value is off (that is, `--disable-lto`). Since
moz.configure is very thorough in passing configure options down into
JS, it dutifully looked at what the default value of `--enable-lto` was
supposed to be, and passed `--disable-lto` to JS's configure.
There's some evidence that we knew our defaulting was a little sketchy:
we'd only attempt cross-language LTO when we were performing the PGO use
phase, and only if the value of `--enable-lto` wasn't explicitly passed.
Which was a fine idea--you don't want to override what the user was
trying to do--but in the case of JS backfired on us: the value *was*
coming from the explicitly-passed command-line option of
`--disable-lto`. So JS didn't enable any kind of LTO, with attendant
consequences.
This problem *didn't* happen before the aforementioned change because we
were explicitly specifying that `--enable-lto=cross` should be passed in
the mozconfig, which ensured that the correct setting was passed into
JS. We were just setting `--enable-lto=cross` for *all* builds, which
was less than desirable.
The easiest way to fix all this is simply to put the
`--enable-lto=cross` setting in the Windows mozconfigs, conditional on
`MOZ_PGO_PROFILE_USE`. That placement captures the intent of the
previous attempt at defaulting, but without the troubles described
above: the option explicitly appears on the command line, and
moz.configure will correctly pass it through to the JS subconfigure.
This also makes our Windows configuration closer to our Linux
configuration (the Linux configuration enables cross-language LTO for
both PGO phases, which is arguably a bug).
Differential Revision: https://phabricator.services.mozilla.com/D41080
--HG--
extra : moz-landing-system : lando
This patch makes BuildEnvironmentNotFoundException a subclass of AttributeError as well, because hasattr in
python3 no longer catches all tracebacks but only AttributeError, and we use both hasattr and
BuildEnvironmentNotFoundException to guard against a handful of buildconfig variables in a few places
where it is OK to not have a buildenvironment.
We also use universal_newlines in real_host in init.configure (since I found
that fix while working on the AttributeError one) so that we get the right string type back from the process call
Lastly this patch also uses BytesIO for calling into a ReducedConfigureSandbox as its stdout and stderr pipes,
This ensures that related code handling the sandbox doesn't complain about getbuffer() missing in StringIO in py3.
Differential Revision: https://phabricator.services.mozilla.com/D36605
--HG--
extra : moz-landing-system : lando
By default, the linker chooses a "generic" 32-bit CPU to optimize for,
and LLVM's "generic" 32-bit CPU model doesn't include some features that
are helpful for performance on microbenchmarks. We explicitly specify a
CPU model to ensure the model we want is selected.
On x86-64, we explicitly force a generically good processor model, even
though the automatically selected one didn't seem to hurt benchmarks.
Differential Revision: https://phabricator.services.mozilla.com/D40479
--HG--
extra : moz-landing-system : lando
This change gives us the ability to selectively turn on cross-language
PGO, just like we have the ability to selectively turn on cross-language LTO.
There is room for things to go wrong here: it's not guaranteed that
`--enable-profile-generate=cross` will always be used with
`--enable-profile-use=cross`. Nothing bad will happen in the sense that
the build will succeed, but it's possible that we miss out on
optimizations on the Rust side. Either we fail to generate profile data
for the Rust code, or the Rust compiler fails to use the profile data.
In the future, we may want to default to cross-language PGO to avoid
these kind of mismatches.
Differential Revision: https://phabricator.services.mozilla.com/D39727
--HG--
extra : moz-landing-system : lando
To do properly checks on LLVM version correspondence between `clang` and
`rustc`, we need information about both of those compilers to be
available. The current placement of the LTO/PGO checks is after we know
something about `clang`, but before we know something about `rustc`.
Therefore we need to move those checks after we've gathered information
about `rustc`.
The PGO bits come along for this bug because the LTO bits depend on
them, and we're going to need the Rust information for cross-language
PGO checks in a different bug. So we might as well move everything all
at once.
Differential Revision: https://phabricator.services.mozilla.com/D39390
--HG--
rename : build/moz.configure/toolchain.configure => build/moz.configure/lto-pgo.configure
extra : moz-landing-system : lando
This change will eventually enable us to cross-check `rustc`'s version
with `clang`'s version when doing cross-language LTO/PGO and avoid
people running into peculiar errors at link time.
Differential Revision: https://phabricator.services.mozilla.com/D39388
--HG--
extra : moz-landing-system : lando
Toolchain path for Windows version is `<NDK ROOT>/toolchains/llvm/prebuilt/windows-x86_64` etc, so it isn't '`winnt`.
So we has to replace `host.kernel.lower()` with `windows`.
Differential Revision: https://phabricator.services.mozilla.com/D39474
--HG--
extra : moz-landing-system : lando
...rather than people running into peculiar crashes running their tests
because functions are pointing at the wrong thing.
It would be more robust to version-check `ld`, but I figure people
wanting to do local cross-language LTO builds is rare enough that
setting an environment variable and rerunning configure is not a huge
hardship.
Differential Revision: https://phabricator.services.mozilla.com/D36742
--HG--
extra : moz-landing-system : lando
...rather than people running into peculiar crashes running their tests
because functions are pointing at the wrong thing.
It would be more robust to version-check `ld`, but I figure people
wanting to do local cross-language LTO builds is rare enough that
setting an environment variable and rerunning configure is not a huge
hardship.
Differential Revision: https://phabricator.services.mozilla.com/D36742
--HG--
extra : moz-landing-system : lando
This build re-uses the PGO profile from the win64 build in the
win64-aarch64-shippable-no-eme part of the aarch64 build. Even though
the profile isn't generated on the smae platform, we still get enough of
a performance win to make this worthwhile.
Note that the pgo_flags() in configure need to be tweaked slightly since
we don't supprt the -fprofile-generate flag for aarch64 (we don't build
the clang_rt.profile lib there). So we always want to return the flags
namespace to make sure we get the use_* versions of flags, which we do
need.
Differential Revision: https://phabricator.services.mozilla.com/D38928
--HG--
extra : moz-landing-system : lando
There is less incentive to keep things building with older versions of
clang for OSX builds, and we're going to require an objective-C feature
that was added in clang 5.
Differential Revision: https://phabricator.services.mozilla.com/D38581
--HG--
extra : moz-landing-system : lando
Actually we set _DEPEND_CFLAGS to both host and target compiler. But if host and target are different compiler type, we may pass invalid option.
Differential Revision: https://phabricator.services.mozilla.com/D38457
--HG--
extra : moz-landing-system : lando
This change is a little bit of a cheat, because of course MSVC doesn't
do cross-language LTO by default, but it seems consistent.
Depends on D33317
Differential Revision: https://phabricator.services.mozilla.com/D33318
--HG--
extra : moz-landing-system : lando
When we build mar, there is no reason not to build signmar as well. It
used to be optional because not all platforms were supported, but they
are now.
... except when building the newly added tools/update-packaging,
which builds the mar tool as a standalone thing, and building signmar
as well causes complications.
Differential Revision: https://phabricator.services.mozilla.com/D36992
--HG--
extra : moz-landing-system : lando
When we build mar, there is no reason not to build signmar as well. It
used to be optional because not all platforms were supported, but they
are now.
Differential Revision: https://phabricator.services.mozilla.com/D36992
--HG--
extra : moz-landing-system : lando
When we build mar, there is no reason not to build signmar as well. It
used to be optional because not all platforms were supported, but they
are now.
--disable-verify-mar is kept to still allow to disable mar verification in
the updater for debugging purpose.
Differential Revision: https://phabricator.services.mozilla.com/D36992
--HG--
extra : moz-landing-system : lando
As it turns out, the version the minidump-stackwalk tasks use (1.31), is
the first version that actually builds the project: 1.30 and earlier
fail because for some reason, cargo wants to read all Cargo.toml files
in the workspace, including unrelated ones, and barfs on features that
weren't supported until 1.31.
Differential Revision: https://phabricator.services.mozilla.com/D37020
--HG--
extra : moz-landing-system : lando
When building Gecko/Android/aarch64 on Windows, `--target` parameter may not be incorrect value. Although `check_compiler`'s `info` is target compiler, clang on Windows is always detected as `clang-cl`, not `clang`.
```
c:/Users/mkato/.mozbuild/clang/bin/clang.exe -E -dM - < /dev/null
...
#define _MSC_VER 1916
```
So even if using clang on Windows, not clang-cl, we should detect as 'clang' correctly
Differential Revision: https://phabricator.services.mozilla.com/D36422
--HG--
extra : moz-landing-system : lando
When host and target are different compiler type, triple for rustc may be incorrect. If target is clang, host is always clang, not using host compiler type.
Example, when host is clang-cl for windows, and target is clang for Android, host's triple for ructc sets `windows-gnu`, not `windows-msvc`.
Differential Revision: https://phabricator.services.mozilla.com/D36421
--HG--
extra : moz-landing-system : lando
When building Gecko/Android/aarch64 on Windows, `--target` parameter may not be incorrect value. Although `check_compiler`'s `info` is target compiler, clang on Windows is always detected as `clang-cl`, not `clang`.
```
c:/Users/mkato/.mozbuild/clang/bin/clang.exe -E -dM - < /dev/null
...
#define _MSC_VER 1916
```
So even if using clang on Windows, not clang-cl, we should detect as 'clang' correctly
Differential Revision: https://phabricator.services.mozilla.com/D36422
--HG--
extra : moz-landing-system : lando
When host and target are different compiler type, triple for rustc may be incorrect. If target is clang, host is always clang, not using host compiler type.
Example, when host is clang-cl for windows, and target is clang for Android, host's triple for ructc sets `windows-gnu`, not `windows-msvc`.
Differential Revision: https://phabricator.services.mozilla.com/D36421
--HG--
extra : moz-landing-system : lando
The SDK headers may not be installed in /usr/include. The usual response
has been to have people run e.g. `open
/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg`
which is not really sustainable.
This makes builds that happen on a macOS host try to detect their SDK
and use that as a default for --with-macos-sdk, which has the side
effect of enabling the SDK version check in that configuration.
Differential Revision: https://phabricator.services.mozilla.com/D36558
--HG--
extra : moz-landing-system : lando
Building host tools on macOS require a macOS SDK, but it's currently
not configurable when cross-compiling for e.g. Android.
Differential Revision: https://phabricator.services.mozilla.com/D36557
--HG--
extra : moz-landing-system : lando
The SDK headers may not be installed in /usr/include. The usual response
has been to have people run e.g. `open
/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg`
which is not really sustainable.
This makes builds that happen on a macOS host try to detect their SDK
and use that as a default for --with-macos-sdk, which has the side
effect of enabling the SDK version check in that configuration.
Differential Revision: https://phabricator.services.mozilla.com/D36558
--HG--
extra : moz-landing-system : lando
Building host tools on macOS require a macOS SDK, but it's currently
not configurable when cross-compiling for e.g. Android.
Differential Revision: https://phabricator.services.mozilla.com/D36557
--HG--
extra : moz-landing-system : lando
With clang-cl and PGO enabled, toolchain.configure automatically turns
on LTO for compatibility with MSVC. However, MOZ_PGO is set for both the
profile-generate and profile-use builds in a 3-tier PGO setup via
imply_option(), but we only want LTO enabled for the profile-use build
(see bug 1483778).
For 1-tier PGO builds, which are still used by local developers, MOZ_PGO
will be set and --enable-profile-generate will be unset, so LTO will be
automatically enabled. The profiledbuild target in make is responsible
for disabling MOZ_LTO on the profile-generate build.
For 3-tier PGO builds, MOZ_PGO will still be set, so we can skip setting
LTO in configure when --enable-profile-generate is set.
Differential Revision: https://phabricator.services.mozilla.com/D34800
--HG--
extra : moz-landing-system : lando
Windows finds llvm-profdata in the PATH, in contrast to Linux or Android
builds that set LLVM_PROFDATA as an environment variable in mozconfigs.
The pgo_profile_path() configure checks should still work in this case.
Differential Revision: https://phabricator.services.mozilla.com/D34799
--HG--
extra : moz-landing-system : lando
Mike Hommey
Nathan Froyd
Justin Wood
Nicholas Nethercote
Makoto Kato
Andreea Pavel
Mike Shal
Simon Sapin
Agi Sferro
Csoregi Natalia
Bogdan Tara
shindli
David Major
Emilio Cobos Álvarez