With libLLVM being a shared library exporting many symbols, all internal
calls using those symbols default to go through the PLT, which is
unnecessary (and costly) overhead. Using -Bsymbolic-functions makes
internal calls go directly to the right place without going through the
PLT.
Differential Revision: https://phabricator.services.mozilla.com/D7029
With libLLVM being a shared library exporting many symbols, all internal
calls using those symbols default to go through the PLT, which is
unnecessary (and costly) overhead. Using -Bsymbolic-functions makes
internal calls go directly to the right place without going through the
PLT.
Differential Revision: https://phabricator.services.mozilla.com/D7029
The cctools-port linker links against libraries from clang (for LTO),
which have different SONAMEs depending on the clang version. Which means
the linker needs to be used along the same version of clang it was built
against. Thus we also make it depend on linux64-clang-7.
But changing the dependency is not enough, cf. bug 1471905, so also
touch its build script, which it turns out, we need to do anyways
because llvm-dsymutil was renamed to dsymutil.
Relatedly, all toolchains that are built using cctools-port need to use
linux64-clang-7 too.
Building compiler-rt 7 with the OSX 10.11 SDK fails because of some
newer APIs being used in compiler-rt for xray, but this is not a feature
we use, so disable that.
Differential Revision: https://phabricator.services.mozilla.com/D6766
--HG--
rename : build/build-clang/clang-6-macosx64.json => build/build-clang/clang-7-macosx64.json
rename : taskcluster/scripts/misc/build-clang-6-linux-macosx-cross.sh => taskcluster/scripts/misc/build-clang-7-linux-macosx-cross.sh
extra : moz-landing-system : lando
The ASan runtime wasn't designed to be unloaded, so pin it in memory.
Differential Revision: https://phabricator.services.mozilla.com/D6943
--HG--
extra : rebase_source : 843fc91b3150f3b264e321ab89bf723a01fc8b3c
Doing so changes the size of the compressed toolchain archive from ~280M
to ~120M, and the decompressed size from ~1500M to ~675M. This will
reduce the overhead of decompression during builds.
As we ship llvm-symbolizer as part of ASan builds, we do need it to
still statically link against LLVM, which we do with a small patch.
With LLVM as a shared library, libLTO, which is used by cctools-port for
the linker, is dynamically linked to LLVM, and the cctools-port
configure script fails to link against libLTO. So we add a -rpath-link
to make it find the LLVM library. This happens to force a rebuild of
cctools-port, but for future cases where we might need a rebuild because
of some clang changes, we add a comment to ease the process, and avoid
a newer cctools-port taking the cache spot of an older one.
Ideally, mac cctools-port would need something similar, but it needs a
mac libLTO.dylib, which is not there anyways (and the mac cctools-port
thus already didn't support LTO).
Also, with LLVM built as a shared library, all its symbols are exported
with a LLVM_x.y version. Combined with -static-libstdc++ that is used
during the clang build, this causes problems (see
https://bugzilla.mozilla.org/show_bug.cgi?id=1492037#c7). But it turns
out things have evolved since -static-libstdc++ has been added to the
clang build script, and things work without now, so remove it (as well
as -static-libgcc).
Differential Revision: https://phabricator.services.mozilla.com/D6117
Doing so changes the size of the compressed toolchain archive from ~280M
to ~120M, and the decompressed size from ~1500M to ~675M. This will
reduce the overhead of decompression during builds.
As we ship llvm-symbolizer as part of ASan builds, we do need it to
still statically link against LLVM, which we do with a small patch.
With LLVM as a shared library, libLTO, which is used by cctools-port for
the linker, is dynamically linked to LLVM, and the cctools-port
configure script fails to link against libLTO. So we add a -rpath-link
to make it find the LLVM library. This happens to force a rebuild of
cctools-port, but for future cases where we might need a rebuild because
of some clang changes, we add a comment to ease the process, and avoid
a newer cctools-port taking the cache spot of an older one.
Ideally, mac cctools-port would need something similar, but it needs a
mac libLTO.dylib, which is not there anyways (and the mac cctools-port
thus already didn't support LTO).
Also, with LLVM built as a shared library, all its symbols are exported
with a LLVM_x.y version. Combined with -static-libstdc++ that is used
during the clang build, this causes problems (see
https://bugzilla.mozilla.org/show_bug.cgi?id=1492037#c7). But it turns
out things have evolved since -static-libstdc++ has been added to the
clang build script, and things work without now, so remove it (as well
as -static-libgcc).
Differential Revision: https://phabricator.services.mozilla.com/D6117
Doing so changes the size of the compressed toolchain archive from ~280M
to ~120M, and the decompressed size from ~1500M to ~675M. This will
reduce the overhead of decompression during builds.
As we ship llvm-symbolizer as part of ASan builds, we do need it to
still statically link against LLVM, which we do with a small patch.
CCTools-port needs to be rebuilt in some cases of clang changes, which
this one of, so touch the script so that it happens.
Differential Revision: https://phabricator.services.mozilla.com/D6117
Not waiting for rc3 or final from llvm because the issue is pretty severe
Differential Revision: https://phabricator.services.mozilla.com/D5124
--HG--
extra : moz-landing-system : lando
As much as it's tempting to use LLD for LTO, it still causes some
subtle problems with the build, and it's still better to keep using
BFD ld for the time being. Doing so requires the gold plugin, which
only requires to pass cmake the directory where the binutils headers
are, and they are part of the gcc toolchain headers.
Differential Revision: https://phabricator.services.mozilla.com/D4896
As much as it's tempting to use LLD for LTO, it still causes some
subtle problems with the build, and it's still better to keep using
BFD ld for the time being. Doing so requires the gold plugin, which
only requires to pass cmake the directory where the binutils headers
are, and they are part of the gcc toolchain headers.
Differential Revision: https://phabricator.services.mozilla.com/D4896
For some reason, clang 6 crashes with a stack overflow on PGO + LTO on
Linux 64 bits. Clang 7 doesn't, but has other problems.
After some bisecting, I found the following:
- r322684 is the first revision that is broken on the release_60 branch.
- that revision is a cherry pick of r322313 from trunk, which is
similarly broken.
- trunk was fixed by r322325, which, funnily enough, predates when
r322313 was cherry-picked.
While the change from r322325 is relatively large, mixing multiple
different changes in a single commit, there also haven't been
significant changes to the same file on trunk since (one macro name
change, one documentation change, and a change related to debug info),
which would tend to indicate the change is not going to break anything,
or at least not more than upgrading to clang 7 would.
The exact part that fixes the issue could probably be found in this
large commit, but I didn't feel like digging into it further considering
the above.
We already copy the 64-bits libraries, but don't copy the 32-bits
libraries, which prevents building for linux32 by default.
Incidentally, this also makes the clang build system build the 32-bits
compiler-rt libraries, allowing e.g. 32-bits PGO.
We already copy the 64-bits libraries, but don't copy the 32-bits
libraries, which prevents building for linux32 by default.
Incidentally, this also makes the clang build system build the 32-bits
compiler-rt libraries, allowing e.g. 32-bits PGO.
https://github.com/rust-lang/rust/issues/52694 is a miscompilation I
found in rust when it uses system llvm 5 or 6, that was fixed 5 months
ago in the llvm rust bundles. This may or may not affect clang, but
considering it was also reported to upstream llvm independently of rust,
it's better to side with caution.
It doesn't affect 3.9, and bug 1478919 got rid of the last use of clang
5 (except for clang-tidy, but that's not used to compile).
The patches come from the llvm trunk from 5 months ago, so they're
already in our clang 7 snapshots.
Windows static analysis builds are still using an old trunk, but are
stuck on bug 1427808. They're "only" for static analysis, though.
--HG--
extra : rebase_source : f4fce69eb7c69b6245518a1bad37e04236c7075b
Instead of clang 4, which they were the last to use, so remove the
clang 4 toolchain.
--HG--
extra : rebase_source : d03a083e9217aeb6c1d2c91decb978426f0e8d1a
svn revert requires a path, and does not take a revision. This isn't an
issue on build machines because we do a fresh checkout every time.
But if you're trying to run build-clang locally, with existing checkouts,
it will:
1) successfully svn update
2) run svn revert, saying "Skipped <rev>"
(except you don't see it because of -q)
3) svn revert returns a successfull eror code
4) patch fails because the file was never reverted and it attempts
to re-apply the patch
Also I think the revert command needs to come first.
MozReview-Commit-ID: 4OfrJNZwJNU
--HG--
extra : rebase_source : b3474e8048b3110f3f5948c3351923c02735ca4d
Note that static analysis was the only remaining user of the 32-bit toolchain, so I've removed win32-clang-cl (or rather, renamed it to win32-clang-cl-st-an).
--HG--
rename : build/build-clang/clang-win32.json => build/build-clang/clang-win32-st-an.json
rename : build/build-clang/clang-win64.json => build/build-clang/clang-win64-st-an.json
rename : taskcluster/scripts/misc/build-clang32-windows.sh => taskcluster/scripts/misc/build-clang32-st-an-windows.sh
rename : taskcluster/scripts/misc/build-clang64-windows.sh => taskcluster/scripts/misc/build-clang64-st-an-windows.sh
I was able to reproduce the failure to apply llvm-dsymutil on the last
mozilla-central revision before debug info was temporarily disabled for
rust, and validated that the problem was gone in clang 4.
After some bisection, r289565 was identified as having fixed the
problem, which, reading the commit message, makes sense.
It was however not possible to simply cherry-pick, because of multiple
code changes between 3.9 and 4. However, apart from the volume of
conflicting changes, it was more or less straightforward to backport.
Interestingly, the patch for r313872 was relying on changes from
r289565, which is why it required a variant specifically for 3.9, but
now we can use the same patch as for other versions. Well, except
there's a small difference in the context, and build-clang.py doesn't
allow fuzz, so we manually edit the patch to remove that line from the
context.
--HG--
extra : rebase_source : de0ab262d401c37c0e9300b0ef7923a07c009d87
I was able to reproduce the failure to apply llvm-dsymutil on the last
mozilla-central revision before debug info was temporarily disabled for
rust, and validated that the problem was gone in clang 4.
After some bisection, r289565 was identified as having fixed the
problem, which, reading the commit message, makes sense.
It was however not possible to simply cherry-pick, because of multiple
code changes between 3.9 and 4. However, apart from the volume of
conflicting changes, it was more or less straightforward to backport.
Interestingly, the patch for r313872 was relying on changes from
r289565, which is why it required a variant specifically for 3.9, but
now we can use the same patch as for other versions. Well, except
there's a small difference in the context, and build-clang.py doesn't
allow fuzz, so we manually edit the patch to remove that line from the
context.
--HG--
extra : rebase_source : cda077132ee499a8ffdc4c88a1160cfa5fd86a97
Before, the build root was not in a Docker cache or volume. With
current Docker works, that meant AUFS. We know AUFS is slow under
I/O load and can cause random failures due to missing data after
writes.
This commit changes the build root to a known Docker volume, which
will be backed by EXT4 and won't have the problems of AUFS.
MozReview-Commit-ID: 6WOH0yednAv
--HG--
extra : rebase_source : bbff0f00f55acdbe068fdf617a7903b8a303c397
32-bit clang-cl.exe was looking specifically for HostX86\x86\link.exe, which doesn't exist in our automation package. Make it look in HostX64\x86 instead.
This is an ugly hack and it would be preferable to just use a 64-bit compiler (bug 1414287).
In a proper VS install, the path to cl.exe looks like:
...\VC\Tools\MSVC\14.11.25503\bin\HostX64\x64\cl.exe
In our automation, the path is just:
...\VC\bin\HostX64\x64\cl.exe
Clang tries to do some sanity-checking to make sure that the cl.exe it finds is the Microsoft compiler and not some other program. But the checks are a little too strict for us, so just look for "bin\Host*\*\cl.exe".
Ideally, we'd backport it to the clang 4 toolchain too, but that results
in silently(!) missing crash symbols for libxul.
--HG--
extra : rebase_source : a3024abf720ea6e9dc0e3271a5aa2ebfae9bebee
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: 554IPMRWgzK
--HG--
extra : rebase_source : 00827d3f6bd705419bc801eb05b543af1ddc274f
shutil.copy2() will fail if the destination directory doesn't exist.
Switch to copy_tree() instead so we don't need to worry about the
error cases of copy2() and copytree().
MozReview-Commit-ID: 3kHfgL57KfX
--HG--
extra : rebase_source : c7335b0c2854d53699dda0f0d2bd9d17b57c4e5d
We're soon going to build multiple versions of clang and gcc for linux,
and we need to differentiate them. Furthermore, there is a need for the
base-toolchains builds to use a fixed version of clang and gcc. So
rename the clang and gcc toolchain jobs to include their version, add
aliases to satisfy all existing jobs, and adjust the base-toolchains
jobs to use the explicit version.
--HG--
rename : build/build-clang/clang-linux64.json => build/build-clang/clang-3.9-linux64.json
rename : taskcluster/scripts/misc/build-gcc-linux.sh => taskcluster/scripts/misc/build-gcc-4.9-linux.sh
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: zehcGJrUQX
--HG--
extra : source : feedcde68c2a54da210f03eb287ab5c862fc982b
extra : intermediate-source : 485d1af7805ad9fa0e701c3571fc1291fbfc6850
The return-empty-string-non-mangled.patch apparently landed in r249437
upstream, so we don't need to carry around our own version. The patch
is still nominally used for building clang on Mac, so I have not removed
it from version control.
Instead of relying on the assumption that a previous run of CMake was
using the same arguments, remove the CMake cache file and re-run it.
This way the script is robust no matter what kind of build directory
existed from before.
Since individual config files have different source repos declared,
it's better to deal with each individual source directory separately.
Also make sure to revert any of the existing changes in each directory
so that attempts to apply patches to the source directory or import
our static analysis checks into clang-tidy are guaranteed to always
succeed.
This revision fixes a clang devirtualization bug and also includes
support for -fbuiltin-module-map, which is useful in making ffmpeg
compile correctly.
clang-cl would normally derive its MSVC emulation bits from the
installed MSVC version, but we don't have an installed MSVC in this
scenario, so we have to use command-line options instead. We use
similar options for Gecko builds.
In a taskcluster world, we cannot used fixed directories, since we don't
know the absolute path of the directory we're building in ahead of time.
(We could pass it in to the build script, or discover it in the script
itself, but that wouldn't really solve the next problem.) This change
does make the builds not reproducible, but as we're using clang-cl
purely for secondary purposes on Windows, rather than for shipping
Firefox binaries (as we would on Mac, say), I don't feel bad about
punting the reproducibility issue down the road a bit.
We cannot depend on a fixed location for cl.exe in a taskcluster world.
We therefore need to make our build-clang.py script accomodate relative
path names for cc/cxx and assume those are binaries that should be
looked up on PATH.
We also need to modify the Linux build script so that the virtualenv is
used to look up the new 'which' package.
We need to rebuild clang with libc++ to get compatible headers for cross
builds. libc++abi is a dependency of libc++, as the build instructions
says [0].
[0] http://libcxx.llvm.org/docs/BuildingLibcxx.html
Ideally, we'd use the tarballs from
http://llvm.org/releases/download.html but I didn't feel like modifying
the script more than I already did to make it work at all (bug 1262735).
The new tarball for Linux was built on
https://tools.taskcluster.net/task-inspector/#LCUn8aEgTBeRJ11a3qTlDQ/0
The new tarball for Mac was built on a loaner, after installing cmake
and ninja, as well as building ld64 127.2 from source because the
installed version would assert while building clang. The latter required
manually adding some missing headers to /usr/include. TSAN was also
disabled because it requires APIs that are not available on the OSX
version on the build slaves (e.g. pthread_introspection_hook_install).
Building clang also required using a mac clang from tooltool, the system
one lacking support for atomics.
Both point to the same path for cc and cxx, but not for gcc_dir, which
makes no sense. That's the only significant difference between both, so
just merge them both, and use the merged file in the taskcluster build
script.
Instead, copy libgcc from the GCC used to build clang at each stage.
When passing --gcc-toolchain, the flag ends up appearing in the output
of llvm-config, and completely defeats the purpose of copying libgcc in
clang/lib/gcc.
Since build-clang.py requires a gcc_dir to be set, and we're using GCC
from there to build clang, we might as well copy libgcc from there
instead of building a fresh GCC. On the taskcluster job building clang,
GCC comes from a tooltool package that is already the result of
build-gcc anyways.
When passing -DCMAKE_C_COMPILER="gcc -flags" to CMake, ninja then tries
to literally call "gcc -flags", not "gcc" "-flags", and a "gcc -flags"
file obviously doesn't exist, so the build fails.
This fixes a regression from bug 1042132. Hopefully, this also doesn't
regress bug 1042132 itself.
Mike Hommey
Coroiu Cristina
Nathan Froyd
David Major
Andi-Bogdan Postelnicu
Brindusan Cristian
Sylvestre Ledru
arthur.iakab
Bogdan Tara
Tom Ritter
Daniel Varga
Christian Holler
Devika Sugathan
Cosmin Sabou
Jacek Caban
Marco Castelluccio
Andreea Pavel
Narcis Beleuzu
Ryan VanderMeulen
Jesse Schwartzentruber
Ciure Andrei
Gregory Szorc
Boris Zbarsky
Wander Lairson Costa
Ting-Yu Chou
Wes Kocher
Ehsan Akhgari
Iris Hsiao
Phil Ringnalda