In order to avoid crashes when we're dumping gcda files, we backport a llvm/compiler-rt patch which fix this issue in adding a critical section around flush.
Differential Revision: https://phabricator.services.mozilla.com/D56559
--HG--
extra : moz-landing-system : lando
What this means is that the sources for clang/llvm are downloaded
separately from the toolchain build (which also means we finally only
download a given version of clang once for all platforms).
In turn, this means the build-clang.py script needs to start with an
existing llvm-project tree, and we choose to make build-clang.py expect
that it's run from the llvm-project root directory.
This also means we don't need to download git for the windows toolchain
task.
Differential Revision: https://phabricator.services.mozilla.com/D40402
Firefox uses multiple processes. It has intentional leaks, and when
running with ASAN, we have suppressions to eliminate those. When running
ASAN builds through CI tests, when Firefox exits, each of the processes
(parent and child) exits and goes through its leaks and when there are
(which is a given), the ASAN runtime runs llvm-symbolizer to symbolicate
and match against suppressions. So each process runs llvm-symbolizer. At
the same time.
Some of the addresses to symbolicate are in libxul. Which contains all
DWARF info, making it a ~1GB monster. Oh, and because you're lucky,
things align perfectly such that libxul size is a multiple of the page
size. That makes llvm-symbolizer pread() the file instead of mmap()ing
it. Did I say there are multiple processes? So suddenly you have n
processes simultaneously allocating and filling 1GB of memory each, on
CI machines that have enough memory for the job they usually run, but
not enough for a sudden rush of n GB.
And things go awry. When you're lucky and the OOM killer didn't take
care of killing the CI entirely, symbolication couldn't happen and the
suppressions are not matched, and leaks are reported.
This all turns out it originates in how llvm-symbolicate chooses between
pread() and mmap(), which turns out is just defaults not being made for
binary files.
Differential Revision: https://phabricator.services.mozilla.com/D16010
--HG--
extra : moz-landing-system : lando
Doing so changes the size of the compressed toolchain archive from ~280M
to ~120M, and the decompressed size from ~1500M to ~675M. This will
reduce the overhead of decompression during builds.
As we ship llvm-symbolizer as part of ASan builds, we do need it to
still statically link against LLVM, which we do with a small patch.
With LLVM as a shared library, libLTO, which is used by cctools-port for
the linker, is dynamically linked to LLVM, and the cctools-port
configure script fails to link against libLTO. So we add a -rpath-link
to make it find the LLVM library. This happens to force a rebuild of
cctools-port, but for future cases where we might need a rebuild because
of some clang changes, we add a comment to ease the process, and avoid
a newer cctools-port taking the cache spot of an older one.
Ideally, mac cctools-port would need something similar, but it needs a
mac libLTO.dylib, which is not there anyways (and the mac cctools-port
thus already didn't support LTO).
Also, with LLVM built as a shared library, all its symbols are exported
with a LLVM_x.y version. Combined with -static-libstdc++ that is used
during the clang build, this causes problems (see
https://bugzilla.mozilla.org/show_bug.cgi?id=1492037#c7). But it turns
out things have evolved since -static-libstdc++ has been added to the
clang build script, and things work without now, so remove it (as well
as -static-libgcc).
Differential Revision: https://phabricator.services.mozilla.com/D6117
Doing so changes the size of the compressed toolchain archive from ~280M
to ~120M, and the decompressed size from ~1500M to ~675M. This will
reduce the overhead of decompression during builds.
As we ship llvm-symbolizer as part of ASan builds, we do need it to
still statically link against LLVM, which we do with a small patch.
With LLVM as a shared library, libLTO, which is used by cctools-port for
the linker, is dynamically linked to LLVM, and the cctools-port
configure script fails to link against libLTO. So we add a -rpath-link
to make it find the LLVM library. This happens to force a rebuild of
cctools-port, but for future cases where we might need a rebuild because
of some clang changes, we add a comment to ease the process, and avoid
a newer cctools-port taking the cache spot of an older one.
Ideally, mac cctools-port would need something similar, but it needs a
mac libLTO.dylib, which is not there anyways (and the mac cctools-port
thus already didn't support LTO).
Also, with LLVM built as a shared library, all its symbols are exported
with a LLVM_x.y version. Combined with -static-libstdc++ that is used
during the clang build, this causes problems (see
https://bugzilla.mozilla.org/show_bug.cgi?id=1492037#c7). But it turns
out things have evolved since -static-libstdc++ has been added to the
clang build script, and things work without now, so remove it (as well
as -static-libgcc).
Differential Revision: https://phabricator.services.mozilla.com/D6117
Calixte Denizet
Mike Hommey
Sylvestre Ledru
Bogdan Tara
Christian Holler
Noemi Erli
Ciure Andrei
Andreea Pavel
Kartikaya Gupta
Cosmin Sabou
Marco Castelluccio