This simplifies the setup for cross clang repacks.
Also use `ninja install` to get all the relevant things in the artifact.
Differential Revision: https://phabricator.services.mozilla.com/D138745
This improves out-of-the-box support for cross-compiles on tier-1 build
platforms, including for asan. This also reduces the number of different
*-cross toolchains we need for CI builds.
The linux clang is now also a repack, like the others, and the insertion
of the wasm compiler-rt is moved to the repack, which also allows to
remove that part of the build-clang.py script.
Differential Revision: https://phabricator.services.mozilla.com/D138749
Instead of building compiler-rt alongside clang, we reuse the clang
repack logic to put compiler-rts in place when building clang stage2
(which requires the profiling runtime).
The repack logic is unified to handle all platforms as well, and now
that compiler-rt is not built in clang, we also need to copy everything
that is not lib/* from the compiler-rt artifacts, so we now also ensure
that they're not diverging between those compiler-rt artifacts (except
for darwin libs, which need to be unified into universal libraries).
And as a bonus, we can also cross-build the mac clang toolchains with
the stage1 clang and the clang runtime for the target platform.
Differential Revision: https://phabricator.services.mozilla.com/D138748
This simplifies the setup for cross clang repacks.
Also use `ninja install` to get all the relevant things in the artifact.
Differential Revision: https://phabricator.services.mozilla.com/D138745
Bug 1752977 attempted to do that, but that part had to be backed out as
a fixup because it broke LTO support for the linker.
To make it work, cctools needs to link against clang's libLTO, which
requires libLTO to be built with the toolchain sysroot. But system GCC
doesn't support that in a nice way, so we switch to system clang. We
explicitly don't use /usr/bin/clang++ because that would use the system
libstdc++ headers instead of the ones from the sysroot, while using
/usr/lib/llvm-11/bin/clang++ uses the ones from the sysroot.
We keep building clang 5.0 with gcc, though, because that one fails to
build with clang 11.
Differential Revision: https://phabricator.services.mozilla.com/D138712
The new task builds a stage 2 compiler and uses it to generate the
profile as stage 3, and the profile is used to feed the now separate
stage 4 clang build. This will allow us to use this same profile to
build mac clangs with PGO.
Differential Revision: https://phabricator.services.mozilla.com/D138511
The new task builds a stage 2 compiler and uses it to generate the
profile as stage 3, and the profile is used to feed the now separate
stage 4 clang build. This will allow us to use this same profile to
build mac clangs with PGO.
Differential Revision: https://phabricator.services.mozilla.com/D138511
The patch conflicts with latest clang trunk changes.
It was necessary back when we started using clang-cl instead of MSVC,
and we needed to fallback to MSVC for sources that clang-cl couldn't
build. Nowadays, with cross-compiles, that fallback can't happen
anyways, and even on native Windows builds we never use cl.exe.
Differential Revision: https://phabricator.services.mozilla.com/D138615
llvmorg-13-init-8182-gc2297544c047 was finally backed out upstream which
means we don't need to revert it or its followups
(llvmorg-14-init-3651-g85ba583eba19 and llvmorg-14-init-3652-gf3c2094d8c11)
Differential Revision: https://phabricator.services.mozilla.com/D138399
This means we can now skip building a 32-bit compiler-rt when building
clang itself, as well as the asan pdb files, which will be taken from
the compiler-rt artifact.
Differential Revision: https://phabricator.services.mozilla.com/D137259
As of bug 1751824, the only remaining use of the 2-stages Windows clang
toolchain is for llvm-profdata in the generate-profile runs.
We don't need that to be built by clang itself (whereas it was desirable
to get compiler-rt from stage 2 before bug 1751824), so we can drop that
build to one stage only.
And because as of bug 1751839 we can start building stage 2 from a
separate stage 1 task, we do that for the Windows clang toolchain, like
we do for the Linux one.
Also, make the win64-clang-toolchain alias point to that stage 1 build,
like for Linux.
Differential Revision: https://phabricator.services.mozilla.com/D137257
This reduces some code duplication and paves the way for modifying how
Mach loads modules.
The plan is to follow this up by making this function take a more fleshed out 'spec'
that maps command names to the files they're defined at and possibly more metadata.
Since this may affect how Mach works internally (e.g. with lazy loading in the roadmap),
it makes sense to move this logic inside the Mach class.
Differential Revision: https://phabricator.services.mozilla.com/D136789
`MOZ_KNOWN_LIVE RefPtr<Foo> mFoo` is not treated as safe because its raw pointer
is referred with operators but they are not checked at handling `MOZ_KNOWN_LIVE`
annotation.
Additionally, when members marked as `MOZ_KNOWN_LIVE` are in the stack, they
are also not treated as safe, but they should be safe in most cases.
With these changes, `HTMLTableEditor.cpp` can get rid of a lot of
`MOZ_KnownLive` method calls.
Differential Revision: https://phabricator.services.mozilla.com/D136122
Back when this was added, clang's default mode might have been c++11,
but it's not c++17, and we're effectively downgrading it (which
incidentally causes problems to build libc++).
Differential Revision: https://phabricator.services.mozilla.com/D137255
It doesn't appear like the breaking changes will affect us.
Note that the taskcluster packages config wasn't updated because
`0.17.0` hasn't yet been deployed to the GitHub releases page.
Differential Revision: https://phabricator.services.mozilla.com/D136947
This avoids needing clang-cl.exe on cross compilations. We could keep
Windows builds on having MIDL do the preprocessing, but that would be
a difference between native and cross builds, and it's better to avoid
that.
Differential Revision: https://phabricator.services.mozilla.com/D136831
This allows to start building other toolchains, compiler runtimes, much
earlier, since we don't have to wait for a complete 4-stage-optimized
compiler to start these, while the output produced by clang is
presumably the same whether it's a stage 1 or a stage 4 clang.
Differential Revision: https://phabricator.services.mozilla.com/D136844
It is currently enabled automatically because libxml2 is detected, but
we want to be sure it's not accidentally disabled.
Differential Revision: https://phabricator.services.mozilla.com/D136840
`extra_targets` was used for the aarch64-linux compiler runtime, which
moved to a separate task in bug 1690757, and `android_targets` was used
for android compiler runtimes, which moved to separate tasks in
bug 1690973.
Using separate tasks is now the prefered way to build compiler runtimes,
os we won't need these options.
Differential Revision: https://phabricator.services.mozilla.com/D136822
It was added for llvm-symbolizer, and at the time, it was required.
However, since [1] (shipped in llvm 12), llvm-symbolizer, msdia140.dll
is not used by default, only when passing the -dia command line option,
which we don't use.
1. c74b427cb2
Differential Revision: https://phabricator.services.mozilla.com/D136821
A bunch of modern packages (`pytest`, `twisted`, `automat`) all need
`attrs==19.2.0` (or newer).
We _could_ bump `attrs` all the way to the modern `21.4.0` version, but
I'd like to defer that upgrade risk, since there's a
lot of backwards-incompatible changes and deprecations. So, lightly bump
it to `19.2.0`.
As part of bumping it, `pytest` is no longer compatible.
The earliest candidate that seems to be compatible is `pytest` 4.6.6,
which boasts in its release notes that it's resolved some deprecation
warnings against `attrs>=19.2.0`.
Once `pytest` was bumped, it needed a newer version of `pluggy`, which
itself has dependencies.
Since we're using hashes in `tox_requirements.txt`, all dependencies
needed to be hashed as well.
Differential Revision: https://phabricator.services.mozilla.com/D135178
- AC_HAVE_FUNCS is an alias to AC_CHECK_FUNCS, so it is covered.
- Nothing uses HAVE___CXA_DEMANGLE, so we don't explictly set it,
although we do need the result of whether __cxa_demangle is supported.
- No moz.build uses MOZ_DEMANGLE_SYMBOLS, so we only `set_define` it.
- We leave dladdr in old-configure because it needs to move along other
dl* things.
- The hotfix for AC_CHECK_FUNC is however not needed for dladdr, so we
remove it.
- We replace the forced HAVE_LOCALECONV on Windows with a check shared
with all platforms.
Differential Revision: https://phabricator.services.mozilla.com/D134858
For some reason, the execution of dbus-launch causes problems after bug
1734934, even though we were executing dbus-launch before. We don't
actually care about valgrind'ing its execution, though, because it's a
system program.
Differential Revision: https://phabricator.services.mozilla.com/D136312
- AC_HAVE_FUNCS is an alias to AC_CHECK_FUNCS, so it is covered.
- Nothing uses HAVE___CXA_DEMANGLE, so we don't explictly set it,
although we do need the result of whether __cxa_demangle is supported.
- No moz.build uses MOZ_DEMANGLE_SYMBOLS, so we only `set_define` it.
- We leave dladdr in old-configure because it needs to move along other
dl* things.
- The hotfix for AC_CHECK_FUNC is however not needed for dladdr, so we
remove it.
- We replace the forced HAVE_LOCALECONV on Windows with a check shared
with all platforms.
Differential Revision: https://phabricator.services.mozilla.com/D134858
Now that command virtualenvs are automatically containing Mach's
sys.path entries, they no longer need to be duplicated.
For compatibility's sake, all "common" requirements have
been moved to be "Mach" requirements. As time goes on
and we know which packages aren't needed for Mach itself,
we can move them back into the common (to commands) requirements
file, and eventually into only the virtualenvs that need them.
Differential Revision: https://phabricator.services.mozilla.com/D132083
Currently, the bootstrap code from configure gets bootstrapped task info
from the taskgraph, and then invokes `mach artifact toolchain` which...
does the same again. That cumulatively wastes a noticeable amount of
time, especially on Windows.
So to avoid repeating this work that the configure side already did,
we do a little more on the configure side (resolution from index to
task-id), and just give a (task-id, artifact path) pair to `mach
artifact toolchain`. The added code to `mach artifact toolchain` is
actually ironically very similar to the code that was removed in
bug 1687594.
And now that mach bootstrap uses the configure code, it benefits from
this change as well.
Differential Revision: https://phabricator.services.mozilla.com/D135311
* the certificates were regenerated using `./mach generate-test-certs`
* the build/pgo/certs/ changes were made using
`./mach python build/pgo/genpgocert.py`
* the zip files in security/manager/ssl/tests/unit/test_signed_apps/ were
updated per the directions in the corresponding moz.build file
* security/manager/ssl/tests/unit/test_cert_override_read.js needed manual
updating
Differential Revision: https://phabricator.services.mozilla.com/D135283
MSYS2 has a slightly different directory structure (binaries are
now under "/usr/bin/" instead of "/bin/"), and we're now plonking
it in `%MOZILLABUILD%\msys2` instead of `%MOZILLABUILD%\msys` so that
MSYS1 files don't interfere with MSYS2 after a pave-over install.
Speaking of pave-over installs: if both `msys2` and `msys` are available,
this patch prefers `msys2`. This is because MozillaBuild installations
with MSYS2 are going to _assume_ they're using MSYS2, and therefore
be most compatible with the versions of packages shipped with MSYS2.
Differential Revision: https://phabricator.services.mozilla.com/D133549
Assuming that the `MOZILLABUILD` environment variable is set, allow
invoking Mach from non-MozillaBuild terminals.
Note that MozillaBuild still needs to be installed, and the
`MOZILLABUILD` environment variable will have to be set.
For future reference: when I tried setting this up with Windows
Store's Python 3.9, I encountered issues when running binaries installed
via `pip`: it would fail with `abort: failed to load Python DLL
python3x.dll`.
Differential Revision: https://phabricator.services.mozilla.com/D133936
Move Python version check as early as possible so that more code can
safely depend on modern behaviour while out-of-date Python versions still
get graceful error messages.
Without this change, Python 2 usages fail on importing `importlib.util`
before the nice "out-of-date version" warning is printed.
Differential Revision: https://phabricator.services.mozilla.com/D134185
As part of this, the shell-script part of `./mach` can be removed,
making it pure Python.
There's a change in `--profile-command` behaviour, though: it now only
profiles the specific command, rather than all of Mach.
This is because _so much of Mach_ has already been run before
CLI arguments are parsed in the Python process.
If a developer wants to profile Mach itself, they can manually run
`python3 -m cProfile -o <file> ./mach ...`
Differential Revision: https://phabricator.services.mozilla.com/D133928
pkgconf is an alternative implementation of pkg-config that is more
cross-platform. It has also become the default on Fedora, so it's not
some random project.
Differential Revision: https://phabricator.services.mozilla.com/D135009
pkgconf is an alternative implementation of pkg-config that is more
cross-platform. It has also become the default on Fedora, so it's not
some random project.
Differential Revision: https://phabricator.services.mozilla.com/D135009
When cmake fails during the clang build, we currently print out the
contents of CMakeOutput.log and CMakeError.log because they may contain
something useful to debug what happened. Unfortunately, because the
treeherder log parser doesn't see the actual error, it reports the
normal error that are part of the logs as being the errors causing the
failure, the first one of which is a check whether malloc/malloc.h is a
thing, which it is not, and that's normal.
So instead of printing them out, we provide the files as artifacts.
Differential Revision: https://phabricator.services.mozilla.com/D134999
Move Python version check as early as possible so that more code can
safely depend on modern behaviour while out-of-date Python versions still
get graceful error messages.
Without this change, Python 2 usages fail on importing `importlib.util`
before the nice "out-of-date version" warning is printed.
Differential Revision: https://phabricator.services.mozilla.com/D134185
As part of this, the shell-script part of `./mach` can be removed,
making it pure Python.
There's a change in `--profile-command` behaviour, though: it now only
profiles the specific command, rather than all of Mach.
This is because _so much of Mach_ has already been run before
CLI arguments are parsed in the Python process.
If a developer wants to profile Mach itself, they can manually run
`python3 -m cProfile -o <file> ./mach ...`
Differential Revision: https://phabricator.services.mozilla.com/D133928
- Avoid the flag selection silently not enabling the new pass manager
when --enable-new-pass-manager is passed explicitly.
- Avoid adding the -fexperimental-new-pass-manager to clang >= 13, which
has it enabled by default. Likewise for the linker flags.
- Remove the force-enable of the new pass manager with clang < 12 on
automation, since we're using version 13 anyways.
- Account for the fact that both lld and ld64 can pass the
-import-hot-multiplier flag to the LTO plugin on mac builds, which
effectively will set it for the first time on mac, and might improve
performance.
Differential Revision: https://phabricator.services.mozilla.com/D134860
We cannot be sure about unknown (that's what we might get on some tier-3
platforms), but if we get e.g. "ld64" while building for Linux, that's
clearly wrong.
Differential Revision: https://phabricator.services.mozilla.com/D134909
Use the SHT_RELR format which significantly improves the size reduction
from elfhack:
total size of .rel.* + .elfhack.* sections
x86 x86_64 android-arm
plain 3532904 10739544 3488888
current-elfhack 1085815 1155994 1042048
relr-elfhack 130219 193552 113840
Differential Revision: https://phabricator.services.mozilla.com/D134756
Port some `sys.path` modifications to the centralized system.
Not only is this cleaner (ad-hoc global modifications are spicy), but
this also avoids "reordering" issues that can creep in during nested
calls, causing virtualenvs to be unnecessarily recreated.
Differential Revision: https://phabricator.services.mozilla.com/D134481
--enable-linker=gold has been supported for a long while, and keeping
enable_gold does add a little extra complexity for not much benefit.
Differential Revision: https://phabricator.services.mozilla.com/D134732
Mike Hommey
Csoregi Natalia
Narcis Beleuzu
Marian-Vasile Laza
Shazib Summar
Thomas Wisniewski
Jan-Erik Rediger
smolnar
Butkovits Atila
Markus Stange
Benjamin Beurdouche
Sylvestre Ledru
Andi-Bogdan Postelnicu
Alex Lopez
Masayuki Nakano
Jan-Ivar Bruaroey
Nick Alexander
Cristian Tuns
Mitchell Hentges
Andreas Pehrson
criss
Iulian Moraru
Andrew Halberstadt
Jesse Schwartzentruber
Agi Sferro
Kagami Sascha Rosylight
Christian Holler
Dana Keeler
Andreea Pavel