The expanded checks in clang 10 made arm64 builds hit CFG crashes on nsXPTCStubBase vtables on startup.
It's not clear why this doesn't happen on x86 builds. Given our current level of support for arm64, I can't really justify investigating this, although I suspect that fixing the underlying issue would be pretty much bug 1483885.
As a get-unblocked stopgap, `-guard:cf,nochecks` in clang 10 gives the same behavior as `-guard:cf` in clang 9.
Differential Revision: https://phabricator.services.mozilla.com/D76216
If we don't do this, configure will happily think the linker kind is
"unknown", which then causes problems downstream.
Differential Revision: https://phabricator.services.mozilla.com/D75300
Add the -X switch to ignore default include path, and tolerate proprocessor failures.
This enables the check to skip a directory that is missing headers, instead of
falling back on the default paths.
Differential Revision: https://phabricator.services.mozilla.com/D73371
We only enabled hardening with an explicit --enable-hardening because we
needed a patch. That patch was applied to upstream clang 8.0.1, so we
can now enable automatically whenever using the right version.
The explicit --enable-hardening was also not applied to win64-aarch64
debug builds, and this indirectly enables it there too, matching other
debug builds. This also avoids breaking debug builds when enabling
winchecksec on cross builds.
Differential Revision: https://phabricator.services.mozilla.com/D68161
--HG--
extra : moz-landing-system : lando
With this and all the previous changes, the necessary mozconfig for
local cross-builds only contains DIA_SDK_PATH, WINDOWSSDKDIR and
--target.
Differential Revision: https://phabricator.services.mozilla.com/D65295
--HG--
extra : moz-landing-system : lando
Depending on the target, the assembler for Windows builds is one of ml,
ml64, or armasm64. Only for the latter currently are we looking for the
assembler with the .exe extension. All of them should be looked for the
same way, and we pick including the extension.
Differential Revision: https://phabricator.services.mozilla.com/D64260
--HG--
extra : moz-landing-system : lando
We're feeding old-configure with compiler and assembler as found by
python configure via ac_cv_* variables (abusing the autoconf cache).
When doing so, and when the assembler was provided via an environment
variable, while python configure resolves the path, old-configure
doesn't get the resolved path, because we're currently passing it the
path that was passed via the environment.
Now pass the resolved path instead.
Differential Revision: https://phabricator.services.mozilla.com/D64258
--HG--
extra : moz-landing-system : lando
After a C++ language version bump, It's possible to wind up in
situations where we are using a new enough compiler version, but the
libstdc++ version in use is not new enough to support new language
features: self-compiled clang with system libraries, clang from `mach
bootstrap` prior to C++ language version bump (and thus including a new
libstdc++ with the boostrapped clang), etc.
Previously, such a situation would mean that things would work fine, and
then start breaking as soon as new library features started to be used.
Let's try to catch the problem earlier, when the update happens, by
verifying that the libstdc++ version is at least as new as the GCC
version we're requiring.
Differential Revision: https://phabricator.services.mozilla.com/D57516
--HG--
extra : moz-landing-system : lando
The doubling of `when` is needed: the `try_run` when ensures that we
only execute the compiler on OS X, and the `@depends` when ensures that
we don't look at the `None` result (which would normally indicate
failure) on non-OS X hosts.
Differential Revision: https://phabricator.services.mozilla.com/D57636
--HG--
extra : moz-landing-system : lando
This change does not fully enable C++17, as we still need standard
library support from some platforms.
Differential Revision: https://phabricator.services.mozilla.com/D54202
--HG--
extra : moz-landing-system : lando
This change does not fully enable C++17, as we still need standard
library support from some platforms.
Differential Revision: https://phabricator.services.mozilla.com/D54202
--HG--
extra : moz-landing-system : lando
We need this for "full" C++17 support (everything is supported, but some
C++17 features still have bugs) and this change also brings Linux into
parity with our Mac requirements.
MANUAL PUSH: build toolchains on inbound to avoid clogging autoland
Differential Revision: https://phabricator.services.mozilla.com/D51450
When running Cargo on Windows, `HOST_LINKER` is required by
`cargo-host-linker.bat`. So we should use windows.configure to set HOST_LINKER
when host OS is Windows and target OS isn't Windows.
Differential Revision: https://phabricator.services.mozilla.com/D37241
--HG--
extra : moz-landing-system : lando
Getting rid of unneeded code and having fewer global defines are both
worthy things.
Differential Revision: https://phabricator.services.mozilla.com/D45726
--HG--
extra : moz-landing-system : lando
A previous patch defined `_LIBCPP_HIDE_FROM_ABI` to the empty string.
This definition worked for normal (opt/debug) builds, but
PGO-instrumented builds fell over with some linker errors. A closer
look at the definition of `_LIBCPP_HIDE_FROM_ABI`:
```
#ifndef _LIBCPP_HIDE_FROM_ABI_PER_TU
# ifndef _LIBCPP_HIDE_FROM_ABI_PER_TU_BY_DEFAULT
# define _LIBCPP_HIDE_FROM_ABI_PER_TU 0
# else
# define _LIBCPP_HIDE_FROM_ABI_PER_TU 1
# endif
#endif
#ifndef _LIBCPP_HIDE_FROM_ABI
# if _LIBCPP_HIDE_FROM_ABI_PER_TU
# define _LIBCPP_HIDE_FROM_ABI _LIBCPP_HIDDEN _LIBCPP_INTERNAL_LINKAGE
# else
# define _LIBCPP_HIDE_FROM_ABI _LIBCPP_HIDDEN _LIBCPP_EXCLUDE_FROM_EXPLICIT_INSTANTIATION
# endif
#endif
```
says that there are two macros involved there: one to set hidden visibility
(which we don't want), and another to exclude the function definition
from explicit instantiation. The semantics of this are not obvious (the
clang documentation only documents this attribute in terms of C++
templates; the particular case we're interested in sets the attribute on
normal function definitions). But roughly, what this does is that it
forces the compiler to see that an equivalent function definition is
provided external to the compilation unit, so it doesn't have to keep
the annotated function definition around.
Differential Revision: https://phabricator.services.mozilla.com/D45073
--HG--
extra : moz-landing-system : lando
We define some libc++-internal macros to avoid conflicts between the way
we do symbol visibility and the way libc++ expects visibility to work.
Newer NDK versions use a newer libc++ which changes the way the macro we
were using works. To enable building with newer libc++ versions, let's
also define macros appropriate to those versions as well.
Differential Revision: https://phabricator.services.mozilla.com/D44510
--HG--
extra : moz-landing-system : lando
Also don't pass universal_newlines to check_cmd_output. That was added
to make python 3 happier when check_cmd_output still returned bytes, to
hint subprocess to return a unicode string, but now that
check_cmd_output does the decoding, that's not needed anymore.
Differential Revision: https://phabricator.services.mozilla.com/D43205
--HG--
extra : moz-landing-system : lando
This change looks a lot more complicated than it actually is.
The motivating issue is that on x86-64 android, the NDK uses gold as its
default linker, and on this platform gold has some bugs in how it handles
`.note.*` sections that lead to `strip` and `objcopy` produce invalid
binaries...or at least invalid binaries according to `elfhack`. When
`elfhack` complains, the build stops, which is suboptimal. Instead, let's
use bfd `ld`, which doesn't have these problems.
Except that when we make the change to provide `--enable-linker` with a
platform-specific default, the lint configure tests complain that the
provided help string is not using the proper format. So we tack on the
sekret `{|}` string to satisfy the linter.
Differential Revision: https://phabricator.services.mozilla.com/D43098
--HG--
extra : moz-landing-system : lando
For some reason, cargo incremental doesn't work as well as it should,
and doesn't perform as well as sccache does. So when building with
sccache, disable cargo incremental. This brought a no-change clobber
build with 100% cache from 3:50 to 2:05 on a beefy AWS instance I was
testing this with.
Differential Revision: https://phabricator.services.mozilla.com/D43188
--HG--
extra : moz-landing-system : lando
Bug 1575135 changed check_cmd_output to return unicode strings, but a
couple places were already trying to do their own decoding, which now
can fail. Remove those.
Interesting the decoding was previously broken on Windows, this
actually fixes it (the output of hg config is not actually utf-8 on
Windows).
Differential Revision: https://phabricator.services.mozilla.com/D43044
--HG--
extra : moz-landing-system : lando
Disable -ftrivial-auto-var-init for DllBLocklistWin.cpp with clang-cl because the file's interceptions happen so early in the main process that the loader hasn't yet resolved the import of memset (used by -ftrivial-auto-var-init) from vcruntime140.dll.
Disable -ftrivial-auto-var-init on Linux32 because it causes some xpcshell test failures.
Differential Revision: https://phabricator.services.mozilla.com/D42273
--HG--
extra : moz-landing-system : lando
Because most calling places in python configure don't actually want to
deal with encodings, although in practical terms they should, make
get_cmd_output handle it itself.
Places that explicitly do want bytes can keep using subprocess directly.
Differential Revision: https://phabricator.services.mozilla.com/D42604
--HG--
extra : moz-landing-system : lando
To do properly checks on LLVM version correspondence between `clang` and
`rustc`, we need information about both of those compilers to be
available. The current placement of the LTO/PGO checks is after we know
something about `clang`, but before we know something about `rustc`.
Therefore we need to move those checks after we've gathered information
about `rustc`.
The PGO bits come along for this bug because the LTO bits depend on
them, and we're going to need the Rust information for cross-language
PGO checks in a different bug. So we might as well move everything all
at once.
Differential Revision: https://phabricator.services.mozilla.com/D39390
--HG--
rename : build/moz.configure/toolchain.configure => build/moz.configure/lto-pgo.configure
extra : moz-landing-system : lando
...rather than people running into peculiar crashes running their tests
because functions are pointing at the wrong thing.
It would be more robust to version-check `ld`, but I figure people
wanting to do local cross-language LTO builds is rare enough that
setting an environment variable and rerunning configure is not a huge
hardship.
Differential Revision: https://phabricator.services.mozilla.com/D36742
--HG--
extra : moz-landing-system : lando
...rather than people running into peculiar crashes running their tests
because functions are pointing at the wrong thing.
It would be more robust to version-check `ld`, but I figure people
wanting to do local cross-language LTO builds is rare enough that
setting an environment variable and rerunning configure is not a huge
hardship.
Differential Revision: https://phabricator.services.mozilla.com/D36742
--HG--
extra : moz-landing-system : lando
This build re-uses the PGO profile from the win64 build in the
win64-aarch64-shippable-no-eme part of the aarch64 build. Even though
the profile isn't generated on the smae platform, we still get enough of
a performance win to make this worthwhile.
Note that the pgo_flags() in configure need to be tweaked slightly since
we don't supprt the -fprofile-generate flag for aarch64 (we don't build
the clang_rt.profile lib there). So we always want to return the flags
namespace to make sure we get the use_* versions of flags, which we do
need.
Differential Revision: https://phabricator.services.mozilla.com/D38928
--HG--
extra : moz-landing-system : lando
There is less incentive to keep things building with older versions of
clang for OSX builds, and we're going to require an objective-C feature
that was added in clang 5.
Differential Revision: https://phabricator.services.mozilla.com/D38581
--HG--
extra : moz-landing-system : lando
Actually we set _DEPEND_CFLAGS to both host and target compiler. But if host and target are different compiler type, we may pass invalid option.
Differential Revision: https://phabricator.services.mozilla.com/D38457
--HG--
extra : moz-landing-system : lando
This change is a little bit of a cheat, because of course MSVC doesn't
do cross-language LTO by default, but it seems consistent.
Depends on D33317
Differential Revision: https://phabricator.services.mozilla.com/D33318
--HG--
extra : moz-landing-system : lando
When building Gecko/Android/aarch64 on Windows, `--target` parameter may not be incorrect value. Although `check_compiler`'s `info` is target compiler, clang on Windows is always detected as `clang-cl`, not `clang`.
```
c:/Users/mkato/.mozbuild/clang/bin/clang.exe -E -dM - < /dev/null
...
#define _MSC_VER 1916
```
So even if using clang on Windows, not clang-cl, we should detect as 'clang' correctly
Differential Revision: https://phabricator.services.mozilla.com/D36422
--HG--
extra : moz-landing-system : lando
When building Gecko/Android/aarch64 on Windows, `--target` parameter may not be incorrect value. Although `check_compiler`'s `info` is target compiler, clang on Windows is always detected as `clang-cl`, not `clang`.
```
c:/Users/mkato/.mozbuild/clang/bin/clang.exe -E -dM - < /dev/null
...
#define _MSC_VER 1916
```
So even if using clang on Windows, not clang-cl, we should detect as 'clang' correctly
Differential Revision: https://phabricator.services.mozilla.com/D36422
--HG--
extra : moz-landing-system : lando
The SDK headers may not be installed in /usr/include. The usual response
has been to have people run e.g. `open
/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg`
which is not really sustainable.
This makes builds that happen on a macOS host try to detect their SDK
and use that as a default for --with-macos-sdk, which has the side
effect of enabling the SDK version check in that configuration.
Differential Revision: https://phabricator.services.mozilla.com/D36558
--HG--
extra : moz-landing-system : lando
Building host tools on macOS require a macOS SDK, but it's currently
not configurable when cross-compiling for e.g. Android.
Differential Revision: https://phabricator.services.mozilla.com/D36557
--HG--
extra : moz-landing-system : lando
The SDK headers may not be installed in /usr/include. The usual response
has been to have people run e.g. `open
/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg`
which is not really sustainable.
This makes builds that happen on a macOS host try to detect their SDK
and use that as a default for --with-macos-sdk, which has the side
effect of enabling the SDK version check in that configuration.
Differential Revision: https://phabricator.services.mozilla.com/D36558
--HG--
extra : moz-landing-system : lando
Building host tools on macOS require a macOS SDK, but it's currently
not configurable when cross-compiling for e.g. Android.
Differential Revision: https://phabricator.services.mozilla.com/D36557
--HG--
extra : moz-landing-system : lando
With clang-cl and PGO enabled, toolchain.configure automatically turns
on LTO for compatibility with MSVC. However, MOZ_PGO is set for both the
profile-generate and profile-use builds in a 3-tier PGO setup via
imply_option(), but we only want LTO enabled for the profile-use build
(see bug 1483778).
For 1-tier PGO builds, which are still used by local developers, MOZ_PGO
will be set and --enable-profile-generate will be unset, so LTO will be
automatically enabled. The profiledbuild target in make is responsible
for disabling MOZ_LTO on the profile-generate build.
For 3-tier PGO builds, MOZ_PGO will still be set, so we can skip setting
LTO in configure when --enable-profile-generate is set.
Differential Revision: https://phabricator.services.mozilla.com/D34800
--HG--
extra : moz-landing-system : lando
Windows finds llvm-profdata in the PATH, in contrast to Linux or Android
builds that set LLVM_PROFDATA as an environment variable in mozconfigs.
The pgo_profile_path() configure checks should still work in this case.
Differential Revision: https://phabricator.services.mozilla.com/D34799
--HG--
extra : moz-landing-system : lando
- `info.type in ('clang-cl', 'clang', 'gcc')` is always true since MSVC
support was removed.
- For some reason, we didn't enforce C++14 on GCC.
Differential Revision: https://phabricator.services.mozilla.com/D34990
--HG--
extra : moz-landing-system : lando
We've been relying on frame pointers being indirectly enabled via things
like --enable-profiling for some time, but this doesn't scale because
some things may want frame pointers while wanting --disable-profiling.
So we move MOZ_FRAMEPTR_FLAGS to python configure and add a new option
to decide whether to enable frame pointers or not.
Differential Revision: https://phabricator.services.mozilla.com/D34117
--HG--
extra : moz-landing-system : lando
Bug 1394825 bumped the minimum version of clang we use on automation,
for the base toolchain jobs, and there's a libclang test for bindgen,
but we should reject the compiler in the first place.
The check to do that works in both C and C++, contrary to the original
3.6 test that was checking a C++ feature ; the test was later changed
for 3.9, using a check that would have worked in both C and C++, but
the C exception was left around. We remove it now.
Differential Revision: https://phabricator.services.mozilla.com/D34083
--HG--
extra : moz-landing-system : lando
The current logic is that if the compiler somehow uses gold rather than
BFD ld by default, we let it, but if it uses BFD ld, we default to lld.
When doing Android builds, the compiler finds the linker in the NDK, and
the default `ld` binary is gold. So we currently end up using fold for
Android builds.
Here, we change the logic such that we use lld when the default linker
the compiler uses is either BFD ld or gold. We can't go with "is not
lld" because the other possible kind, ld64, is what we actually want to
use on mac, since lld doesn't support mach-o fully just yet.
Differential Revision: https://phabricator.services.mozilla.com/D34040
--HG--
extra : moz-landing-system : lando
The current logic is that if the compiler somehow uses gold rather than
BFD ld by default, we let it, but if it uses BFD ld, we default to lld.
When doing Android builds, the compiler finds the linker in the NDK, and
the default `ld` binary is gold. So we currently end up using gold for
Android builds.
Here, we change the logic such that we use lld when the default linker
the compiler uses is either BFD ld or gold. We can't go with "is not
lld" because the other possible kind, ld64, is what we actually want to
use on mac, since lld doesn't support mach-o fully just yet.
Differential Revision: https://phabricator.services.mozilla.com/D34040
--HG--
extra : moz-landing-system : lando
The current setup for bindgen relies on either finding clang/libclang
from the output of llvm-config, or from the paths given via the
configure flags --with-clang-path/--with-libclang-path.
One _very_ common problem is that the llvm-config we end up using does
not correspond to the clang used for compilation, which has some
undesirable side effect, like failing to build.
So instead of relying on llvm-config, we do the following:
- when the compiler is clang, we just use that
- when the compiler is clang-cl, we use clang from the same directory
- otherwise, we either try to find clang in PATH, or rely on
--with-clang-path.
Once clang is found, we try to deduce the location of the corresponding
libclang via the output of `clang -print-search-dirs`, or rely on
--with-libclang-path.
Differential Revision: https://phabricator.services.mozilla.com/D33241
--HG--
extra : moz-landing-system : lando
This change is a little bit of a cheat, because of course MSVC doesn't
do cross-language LTO by default, but it seems consistent.
Depends on D33317
Differential Revision: https://phabricator.services.mozilla.com/D33318
--HG--
extra : moz-landing-system : lando
This form of instrumentation is more like our other platforms, and also
opens the possibility of interacting properly with Rust IR-level PGO.
Differential Revision: https://phabricator.services.mozilla.com/D31133
--HG--
extra : moz-landing-system : lando
e10s profiling or IR-based PGO instrumentation will both produce
multiple `.profraw` files that need to be handled in some way. Since
clang's `-fprofile-generate` option takes a directory, it seems fitting
to make `--with-pgo-profile-path` mirror that by taking a directory, and
letting `merge_profdata.py` deal with whatever files it might find in
said directory.
Differential Revision: https://phabricator.services.mozilla.com/D32389
--HG--
extra : moz-landing-system : lando
We've not been checking the clang-cl version in use. This lack of
checking is bad, for a couple of reasons:
* Released versions of clang-cl differ drastically in their robustness;
* Only the most recent version of clang-cl supports aarch64.
We should check for a minimum version of clang-cl, just like our other
supported compilers. As a bonus, we can then start depending on
features that we know appear in the particular minimum clang-cl
version. (The current patch is motivated by `/clang:` command-line
support, but one could pick other things.)
Differential Revision: https://phabricator.services.mozilla.com/D30723
--HG--
extra : moz-landing-system : lando
This option is for performing LTO between C++ code and Rust code. The
actual build pieces for Rust code are coming in the next patch.
Differential Revision: https://phabricator.services.mozilla.com/D28508
--HG--
extra : moz-landing-system : lando
We already do this for the compiler that we detect via
toolchain.configure...but as the comment in `basic_bindgen_cflags`
alludes to, we don't actually use that compiler here. We use a much
more bare-bones compiler, so we have to add the necessary flags manually.
We need to make `--with-macos-sdk` a `js_option` so that uses of bindgen
in the JS engine will receive the proper flags.
Differential Revision: https://phabricator.services.mozilla.com/D30239
--HG--
extra : moz-landing-system : lando
Some media libraries use gas syntax in their assembly files. Rather than
converting these arm assembly syntax files for aarch64, we can use clang-cl
to build them directly.
Differential Revision: https://phabricator.services.mozilla.com/D27785
--HG--
extra : moz-landing-system : lando
We moved fuzzing options out of `toolkit/moz.configure` into `js`'s
configure a while back, but we seem to have snuck some fuzzing-related
options into `toolchain.configure` in the interim. But we can't make
the `toolchain.configure` bits depend on the `js` bits; let's just put
everything in `toolchain.configure`.
Differential Revision: https://phabricator.services.mozilla.com/D28084
--HG--
extra : moz-landing-system : lando
Bug 1500504 added a version check for the SDK, but it only does the
check if --with-macos-sdk is used. We should also check the version when
using the default SDK.
Note that this means we now set MACOS_SDK_DIR to be the default SDK even
if it wasn't set explicitly from --with-macos-sdk
Differential Revision: https://phabricator.services.mozilla.com/D17727
--HG--
extra : moz-landing-system : lando
Bug 1500504 added a version check for the SDK, but it only does the
check if --with-macos-sdk is used. We should also check the version when
using the default SDK.
Note that this means we now set MACOS_SDK_DIR to be the default SDK even
if it wasn't set explicitly from --with-macos-sdk
Differential Revision: https://phabricator.services.mozilla.com/D17727
--HG--
extra : moz-landing-system : lando
The file is essentially limited to ARM flag tweaks, so we make the
corresponding python configure code ARM-only. In large parts, the
python code sets things up similarly, although in a few places we try to
be smarter about than the current autoconf-based code.
The resulting config.status have been verified to only have
insignificant differences on both arm and aarch64 android builds.
Depends on D24000
Differential Revision: https://phabricator.services.mozilla.com/D24001
--HG--
extra : moz-landing-system : lando
See https://bugs.llvm.org/show_bug.cgi?id=38944. clang thought it could safely leave out type information for some of our classes, but that information would have been helpful for debugging. The flag in this patch turns off the debug-info limiting.
Differential Revision: https://phabricator.services.mozilla.com/D24073
--HG--
extra : moz-landing-system : lando
The file is essentially limited to ARM flag tweaks, so we make the
corresponding python configure code ARM-only. In large parts, the
python code sets things up similarly, although in a few places we try to
be smarter about than the current autoconf-based code.
The resulting config.status have been verified to only have
insignificant differences on both arm and aarch64 android builds.
Depends on D24000
Differential Revision: https://phabricator.services.mozilla.com/D24001
--HG--
extra : moz-landing-system : lando
The only place we'd need the compat libraries would be for host
binaries, and those shouldn't be a problem given that our system images
are new enough.
Differential Revision: https://phabricator.services.mozilla.com/D22873
--HG--
extra : moz-landing-system : lando
It's occasionally useful to know what versions are being discovered for
these variables. It's also convenient for logs from automation to
contain all the relevant information, rather than setting up a loaner
and running all the commands yourself.
Differential Revision: https://phabricator.services.mozilla.com/D22385
--HG--
extra : moz-landing-system : lando
Bogdan Tara
Mike Hommey
David Major
Nathan Froyd
Adam Gashlin
Stefan Hindli
Jesse Schwartzentruber
Oana Pop Rus
Stephen A Pohl
Andi-Bogdan Postelnicu
Andreea Pavel
Tyson Smith
Mihai Alexandru Michis
Chris Manchester
Ricky Stewart
Csoregi Natalia
Cosmin Sabou
Gabriele Svelto
Tom Ritter
Razvan Maries
Makoto Kato
Christian Holler
Marco Castelluccio
Dorel Luca
Sylvestre Ledru
Emilio Cobos Álvarez
Chris Peterson
Cameron Kaiser
Mike Shal
Gijs Kruitbosch
Coroiu Cristina
Kartikaya Gupta
Dan Minor
Bobby Holley
Mark Banner
Eric Rahm
Gurzau Raul
Gian-Carlo Pascutto
Thomas Daede
Nathan Froyd