The cctools-port linker links against libraries from clang (for LTO),
which have different SONAMEs depending on the clang version. Which means
the linker needs to be used along the same version of clang it was built
against. Thus we also make it depend on linux64-clang-7.
But changing the dependency is not enough, cf. bug 1471905, so also
touch its build script, which it turns out, we need to do anyways
because llvm-dsymutil was renamed to dsymutil.
Relatedly, all toolchains that are built using cctools-port need to use
linux64-clang-7 too.
Building compiler-rt 7 with the OSX 10.11 SDK fails because of some
newer APIs being used in compiler-rt for xray, but this is not a feature
we use, so disable that.
Differential Revision: https://phabricator.services.mozilla.com/D6766
Bug 1282866 removed the only configuration where it did something.
Since then, using it only leads to a configure error.
Differential Revision: https://phabricator.services.mozilla.com/D7691
--HG--
extra : moz-landing-system : lando
We need to sign parts of the contents of the archives, so the mar's that we
ship get built as part of the repackage task. Thus, there is no reason to also
create and upload as part of the build, just to throw them away.
Differential Revision: https://phabricator.services.mozilla.com/D6213
--HG--
extra : moz-landing-system : lando
The cctools-port linker links against libraries from clang (for LTO),
which have different SONAMEs depending on the clang version. Which means
the linker needs to be used along the same version of clang it was built
against. Thus we also make it depend on linux64-clang-7.
But changing the dependency is not enough, cf. bug 1471905, so also
touch its build script, which it turns out, we need to do anyways
because llvm-dsymutil was renamed to dsymutil.
Relatedly, all toolchains that are built using cctools-port need to use
linux64-clang-7 too.
Building compiler-rt 7 with the OSX 10.11 SDK fails because of some
newer APIs being used in compiler-rt for xray, but this is not a feature
we use, so disable that.
Differential Revision: https://phabricator.services.mozilla.com/D6766
--HG--
rename : build/build-clang/clang-6-macosx64.json => build/build-clang/clang-7-macosx64.json
rename : taskcluster/scripts/misc/build-clang-6-linux-macosx-cross.sh => taskcluster/scripts/misc/build-clang-7-linux-macosx-cross.sh
extra : moz-landing-system : lando
This incorporates some fixes for ccache, icecream, and a better display
output (decreasing count instead of an increasing count). It may also
let us experiment with incremental rust support, though I think that
feature still needs some more work.
MozReview-Commit-ID: 7zPGiL9Ob6N
Differential Revision: https://phabricator.services.mozilla.com/D7152
--HG--
extra : moz-landing-system : lando
Some build flags are being passed by the build system: they're passed in a text
file called extra-bindgen-flags.in that's filled at configure time.
Other flags have to be inferred from the current target/host combination, in
Cranelift's build script directly. This is mostly cargo-culted from the
ServoBindings.toml file, and should probably be merged in the build system at
some point.
Some Windows-specific adjustments were needed to provide access to libclang for
bindgen support, by adding clang-cl to the plain Spidermonkey Windows builds.
--HG--
extra : rebase_source : 0bda40b1d1eb38c2657593f094c951013711d00a
extra : histedit_source : aad930a5f9099e299d385ae4de2deb81aed9b6d5
The ASan runtime wasn't designed to be unloaded, so pin it in memory.
Differential Revision: https://phabricator.services.mozilla.com/D6943
--HG--
extra : rebase_source : 843fc91b3150f3b264e321ab89bf723a01fc8b3c
With libLLVM being a shared library exporting many symbols, all internal
calls using those symbols default to go through the PLT, which is
unnecessary (and costly) overhead. Using -Bsymbolic-functions makes
internal calls go directly to the right place without going through the
PLT.
Differential Revision: https://phabricator.services.mozilla.com/D7029
With libLLVM being a shared library exporting many symbols, all internal
calls using those symbols default to go through the PLT, which is
unnecessary (and costly) overhead. Using -Bsymbolic-functions makes
internal calls go directly to the right place without going through the
PLT.
Differential Revision: https://phabricator.services.mozilla.com/D7029
This extracts the current logic for finding nodejs into its own module in mozbuild. Configure and ESLint then use it.
For ESLint, this will change the first location it looks for nodejs to be the .mozbuild directory.
Differential Revision: https://phabricator.services.mozilla.com/D6430
--HG--
extra : moz-landing-system : lando
This change prevents javascript:-URLs from being passed at the command
line. This restriction was already applied to every URL but the first.
Even the first URL did not result in any visible effect when Firefox is
started. Yet somehow the PGO profiler script managed to rely on it.
This commit ensures that "javascript:" URLs are not activated regardless of
position in the command line, and switches to a data:-URL for the PGO script to
achieve the (previously) desired effect of quitting the browser on startup.
Depends on D4929
Differential Revision: https://phabricator.services.mozilla.com/D5958
--HG--
extra : moz-landing-system : lando
This patch gets us building an updater binary that always embeds the dep certificates (instead of release or nightly), and builds a new tests package that includes it.
This was originally D5900, but that was backed out due to busting artifact builds. I've fixed that by removing the Makefile that Ted pointed out is unnecessary.
Differential Revision: https://phabricator.services.mozilla.com/D6418
--HG--
extra : moz-landing-system : lando
ParmVarDecl being a subclass of VarDecl, using two matchers then caused
ScopeChecker::check to be called twice for ParmVarDecl nodes, once for
each match. But the code in ScopeCheck::check is written with the
assumption that it's called only once for such nodes.
Somehow, this didn't cause problems with clang up to version 6, but
makes the plugin spuriously warn about already_AddRefed not being used
as temporaries when used as argument in function declarations, with
clang 7.
Differential Revision: https://phabricator.services.mozilla.com/D6360
--HG--
extra : moz-landing-system : lando
Doing so changes the size of the compressed toolchain archive from ~280M
to ~120M, and the decompressed size from ~1500M to ~675M. This will
reduce the overhead of decompression during builds.
As we ship llvm-symbolizer as part of ASan builds, we do need it to
still statically link against LLVM, which we do with a small patch.
With LLVM as a shared library, libLTO, which is used by cctools-port for
the linker, is dynamically linked to LLVM, and the cctools-port
configure script fails to link against libLTO. So we add a -rpath-link
to make it find the LLVM library. This happens to force a rebuild of
cctools-port, but for future cases where we might need a rebuild because
of some clang changes, we add a comment to ease the process, and avoid
a newer cctools-port taking the cache spot of an older one.
Ideally, mac cctools-port would need something similar, but it needs a
mac libLTO.dylib, which is not there anyways (and the mac cctools-port
thus already didn't support LTO).
Also, with LLVM built as a shared library, all its symbols are exported
with a LLVM_x.y version. Combined with -static-libstdc++ that is used
during the clang build, this causes problems (see
https://bugzilla.mozilla.org/show_bug.cgi?id=1492037#c7). But it turns
out things have evolved since -static-libstdc++ has been added to the
clang build script, and things work without now, so remove it (as well
as -static-libgcc).
Differential Revision: https://phabricator.services.mozilla.com/D6117
Doing so changes the size of the compressed toolchain archive from ~280M
to ~120M, and the decompressed size from ~1500M to ~675M. This will
reduce the overhead of decompression during builds.
As we ship llvm-symbolizer as part of ASan builds, we do need it to
still statically link against LLVM, which we do with a small patch.
With LLVM as a shared library, libLTO, which is used by cctools-port for
the linker, is dynamically linked to LLVM, and the cctools-port
configure script fails to link against libLTO. So we add a -rpath-link
to make it find the LLVM library. This happens to force a rebuild of
cctools-port, but for future cases where we might need a rebuild because
of some clang changes, we add a comment to ease the process, and avoid
a newer cctools-port taking the cache spot of an older one.
Ideally, mac cctools-port would need something similar, but it needs a
mac libLTO.dylib, which is not there anyways (and the mac cctools-port
thus already didn't support LTO).
Also, with LLVM built as a shared library, all its symbols are exported
with a LLVM_x.y version. Combined with -static-libstdc++ that is used
during the clang build, this causes problems (see
https://bugzilla.mozilla.org/show_bug.cgi?id=1492037#c7). But it turns
out things have evolved since -static-libstdc++ has been added to the
clang build script, and things work without now, so remove it (as well
as -static-libgcc).
Differential Revision: https://phabricator.services.mozilla.com/D6117
This commit updates submit_telemetry_data.py to send data
to the Telemetry pipeline. The script assumes the presence
of a "telemetry" directory within the statedir, and an
"outgoing" directory within the "telemetry" directory (otherwise
there is no data to submit). The script will create a
"submitted" directory and "telemetry.log" file if absent,
making the assumption that this is the first build telemetry
submission for that user. UUID values for submitted data points
are seeded from the filename, without the ".json" suffix.
Differential Revision: https://phabricator.services.mozilla.com/D5687
--HG--
extra : moz-landing-system : lando
This patch gets us building an updater binary that always embeds the dep certificates (instead of release or nightly), and builds a new tests package that includes it.
I had a lot of trouble getting the test package generated correctly due to the fact that things in "_tests" won't be included for test packages that aren't "common". My fix for that isn't ideal - I'm open to something better.
Differential Revision: https://phabricator.services.mozilla.com/D5900
--HG--
extra : moz-landing-system : lando
Doing so changes the size of the compressed toolchain archive from ~280M
to ~120M, and the decompressed size from ~1500M to ~675M. This will
reduce the overhead of decompression during builds.
As we ship llvm-symbolizer as part of ASan builds, we do need it to
still statically link against LLVM, which we do with a small patch.
CCTools-port needs to be rebuilt in some cases of clang changes, which
this one of, so touch the script so that it happens.
Differential Revision: https://phabricator.services.mozilla.com/D6117
BUILD_TOOLS was only ever used for things that another variable provides
equally well. Removing BUILD_TOOLS means that we can remove win_srcdir
and WIN_TOP_SRC as well.
All but one of the current uses of DEFFILE use `SRCDIR + '/file.def'` to
get a srcdir-relative path anyway, and the other one wants an
objdir-relative path, so using Path makes everything clearer.
This makes it more straightforward to translate the paths for the WSL
build.
- We forcefully remove annotations from the AST so that they don't end
up impacting codegen.
- We change the API such that we use identifiers instead of strings,
reducing the chances of typo errors.
Differential Revision: https://phabricator.services.mozilla.com/D5493
--HG--
extra : moz-landing-system : lando
We were already always enabling it on builds with frame pointers on
Android. We now extend this to builds without frame pointers, helping
with client-side stacktracing for crashes.
It's also the default on many platforms, such as x86-64, whether frame
pointer are enabled or not and on x86 with GCC. Somehow, it's not the
case for x86 with clang, so this makes clang builds the same as GCC
builds in that regard.
On ARM, we were always omitting frame pointers because of a GCC bug, but
we've not been using GCC for Android builds for a long time (and the GCC
bug was actually fixed in 4.7 and backported to 4.6, making it fixed in
all versions of GCC we support).
Differential Revision: https://phabricator.services.mozilla.com/D6110
--HG--
extra : moz-landing-system : lando
This patch makes it so we write the sccache log directly to the artifact
directory, so that it will be uploaded even if the build fails. It also makes
the log slightly more verbose. Both of these should help with diagnosing
sccache failures in CI.
The sccache log will no longer be explicitly gzip compressed, but some
Taskcluster client implementations will store it as gzip compressed.
Differential Revision: https://phabricator.services.mozilla.com/D6187
--HG--
extra : moz-landing-system : lando
When checking whether the new relocations sizes are going to be a win, we
need to account for the fact that there are non-elfhacked relocation
remaining.
Differential Revision: https://phabricator.services.mozilla.com/D5837
--HG--
extra : moz-landing-system : lando
Now that we ship builds using clang on all platforms, pick it during
configure. It is still possible to opt-in to building other compilers by
setting CC/CXX (or even only CC) to the desired compiler.
Depends on D5829
Differential Revision: https://phabricator.services.mozilla.com/D5637
--HG--
extra : moz-landing-system : lando
Before this change, we'd derive a host compiler and handle things as if
HOST_CC/HOST_CXX had been passed. With this change, we change the list
of compilers that are tested with check_prog instead.
Depends on D5828
Differential Revision: https://phabricator.services.mozilla.com/D5829
--HG--
extra : moz-landing-system : lando
Currently, when building for Android x86-64 on Linux x86-64, we drop the
'--target' flag, which causes the build to fail. This patch adds a check
for OS mismatch, so we keep the '--target' flag in this situation.
Differential Revision: https://phabricator.services.mozilla.com/D4483
It's nicer to have all that logic in one place, and to be able to common
up the Unix-y flags setting. The Makefile constructs in string values
is gross, but it's no worse than we had before.
Add extra preferences for the profileserver script on-top of
the ones as inheritted from unittests.
--HG--
extra : rebase_source : 02d3297134e82c2425fc4c6c939cc5d0cc4a2c13
extra : histedit_source : 07f699b0ff8bec2a6eda7705a0fa13fb3b352f98
Currently mozconfig.cache overrides a few build options for sccache.
This patch moves them into toolchain.configure so that the build system
will set them properly when sccache is in use. Additionally,
{CC,CXX}_WRAPPER are set in config.mk, so just avoid setting them when
sccache is in use.
Now that Linux PGO builds also do LTO and all the Linux builds use
clang, there's not much use for separate LTO builds.
Differential Revision: https://phabricator.services.mozilla.com/D5632
Currently, sccache doesn't work at all with clang-cl (PGO or not, bug
1476604), and newer versions of sccache that do the right thing for PGO
flags with clang have other problems (see bug 1484897).
The benefit of sccache on PGO+LTO builds is however limited (the
profile-use half of the build doesn't benefit at all, and since it does
LTO as well, it's the most time-consuming part of the build), so in
order not to further block switching Linux builds to clang, let's just
disable sccache on PGO builds for now (the earlier we can switch to
clang, the more time it leaves to catch any problem that causes).
Differential Revision: https://phabricator.services.mozilla.com/D5509
--HG--
extra : moz-landing-system : lando
Nothing is using the xpt module anymore, which means we can remove it,
as well as the runtests.py script that runs its test, and the
integration of those tests in the build system.
Depends on D5221
Differential Revision: https://phabricator.services.mozilla.com/D5223
--HG--
extra : moz-landing-system : lando
Not waiting for rc3 or final from llvm because the issue is pretty severe
Differential Revision: https://phabricator.services.mozilla.com/D5124
--HG--
extra : moz-landing-system : lando
Last attempt, a few years ago, blatantly failed because nautilus (the
GNOME file manager) can't start PIE executables, which look like shared
libraries, and that it thus considers not being executables.
Downstreams don't actually have the problem, because users won't be
launching Firefox from a file manager, but for mozilla.org builds, it is
a problem because users would download, then extract, and then likely
try to run the Firefox executable from a file manager.
So for mozilla.org builds, we still need to find a way around the
nautilus problem.
A .desktop file could be a solution, but .desktop files have not
actually been designed for this use case, which leads to:
- having to use an awful one-liner shell wrapper to derive the path
to the executable from that of the .desktop file,
- not even being able to associate an icon,
- the .desktop file not being copiable to a location where .desktop
files would normally go, because it would then fail to find the
executable.
Another possibility is to go back to using a shell wrapper, but that's
not entirely appealing.
What we chose here is similar, where we have a small `firefox` wrapper
that launches the real `firefox-bin` (which is still leftover from those
old times where we had a shell wrapper, for reasons).
The small `firefox` wrapper is a minimalist C executable that just
finds the path to the `firefox-bin` executable and executes it with the
same args it was called with. The wrapper is only enabled when the
MOZ_NO_PIE_COMPAT environment variable is set, which we only take into
account on Linux. The variable is only really meant to be used for
mozilla.org builds, for the nautilus problem. Downstreams will just pick
the default, which is changed to build PIE.
On other platforms, PIE was already enabled by default, so we just
remove the --enable-pie configure flag.
Differential Revision: https://phabricator.services.mozilla.com/D5109
When building executables as PIE, and because we use -Bsymbolic, which
symbols are exported from an executable varies based on the libraries
it's directly linked against, to fulfil their symbol needs. So when a
library depends on e.g. ASAN runtime symbols, and the linker finds that,
it will keep those ASAN symbols in the executable for the library. And
drop the other, unused symbols.
But when the executable then dlopen()s a library (e.g. shlibsign loading
libfreebl) that uses another set of ASAN symbols, including symbols that
none of the direct dependencies of the executable need, dlopen() fails
because of the missing symbols.
It's not currently an apparent problem because we don't enable PIE, and
we build Gecko executables with -rdynamic already (for mozjemalloc). But
we don't build non-Gecko executables this way (like shlibsign).
Differential Revision: https://phabricator.services.mozilla.com/D5108
Bug 1487330 made mozconfig.asan use mozconfig.linux, but while that
worked out fine, it turns out that we do have mac builds using
e.g. mozconfig.asan, and that those can break when doing linux-specific
changes to mozconfig.linux.
--HG--
rename : build/unix/mozconfig.linux => build/unix/mozconfig.unix
As per bug 1487622, annotations have side effects on codegen, one of
which changes a leak signature that the wpt harness doesn't match
against its whitelist anymore.
Interestingly, while looking at the issue, I found multiple moz_*
annotations that weren't actually looked for by the clang plugin,
making them useless. Even more interestingly, removing them changes
codegen in such a way that the leak signature mentioned above is not
altered anymore.
Differential Revision: https://phabricator.services.mozilla.com/D5000
As described in c2b5cf7bde83, it is still preferable to build with BFD
ld when doing clang LTO, and one of the reasons we defaulted to lld in
the first place is that we didn't have the LLVM gold plugin on
automation, which, as of bug 1488307, we now have.
Differential Revision: https://phabricator.services.mozilla.com/D4987
At least, when that combination doesn't work. This requires passing the
LTO flags to the compiler invocations for the test, which in turn
requires actually calling the linker directly on Windows, instead of
directly through the compiler, as when actually building.
Differential Revision: https://phabricator.services.mozilla.com/D4739
--HG--
extra : moz-landing-system : lando
As much as it's tempting to use LLD for LTO, it still causes some
subtle problems with the build, and it's still better to keep using
BFD ld for the time being. Doing so requires the gold plugin, which
only requires to pass cmake the directory where the binutils headers
are, and they are part of the gcc toolchain headers.
Differential Revision: https://phabricator.services.mozilla.com/D4896
Back before expandlibs was removed in bug 1429875, not using list files
was supported, but nowadays, it just leads to errors like the following:
*** No rule to make target 'libfake_so.list', needed by 'libfake.so'. Stop.
We might as well error out during configure, rather than much later
during the build.
Differential Revision: https://phabricator.services.mozilla.com/D4738
All the Linux builds using GCC uses the binutils bundled with GCC. This
gives us some leeway to update the binutils used for clang builds (using
the binutils toolchain as of bug 1486998) separately.
Since we only ship builds using GCC, we're more free to upgrade
binutils for clang builds, without worrying about the next merge.
This upgrades to the last released version of binutils, and applies the
patch from https://sourceware.org/bugzilla/show_bug.cgi?id=23591 on top,
so that asan fuzzing builds don't fail.
The GPG key used to sign the upstream tarball is unfortunately not
connected to the web of trust. I verified the contents matched what's in
the Debian archive (which has a different tarball, because some files
are removed/modified in Debian for license reasons ; there were no
differences besides those).
Differential Revision: https://phabricator.services.mozilla.com/D4748
--HG--
extra : moz-landing-system : lando
Last time it was updated is bug 1436208, and the crashes we patched it
for back then has been fixed upstream a few months later.
For some reason, they renamed the executable from llvm-dsymutil to
dsymutil.
Differential Revision: https://phabricator.services.mozilla.com/D4741
--HG--
extra : moz-landing-system : lando
Those mozconfigs do things in common that are already in
mozconfig.linux. Let's just use that.
This will make the build have the binutils toolchain in PATH rather than
passing it to CC/CXX, which is better because it also makes the build
use tools such as ar, ranlib, objcopy from the binutils toolchain,
rather than the old system ones.
Differential Revision: https://phabricator.services.mozilla.com/D4649
Back before expandlibs was removed in bug 1429875, not using list files
was supported, but nowadays, it just leads to errors like the following:
*** No rule to make target 'libfake_so.list', needed by 'libfake.so'. Stop.
We might as well error out during configure, rather than much later
during the build.
Differential Revision: https://phabricator.services.mozilla.com/D4738
As much as it's tempting to use LLD for LTO, it still causes some
subtle problems with the build, and it's still better to keep using
BFD ld for the time being. Doing so requires the gold plugin, which
only requires to pass cmake the directory where the binutils headers
are, and they are part of the gcc toolchain headers.
Differential Revision: https://phabricator.services.mozilla.com/D4896
Mike Hommey
Ciure Andrei
Csoregi Natalia
Guillaume GARDET
Kartikaya Gupta
shindli
Noemi Erli
Mike Shal
Gurzau Raul
Brindusan Cristian
Nathan Froyd
Tom Prince
Coroiu Cristina
Chris Manchester
Benjamin Bouvier
David Major
Narcis Beleuzu
Mark Banner
Andi-Bogdan Postelnicu
Rob Wu
Sylvestre Ledru
Ben Hearsum
Thomas P.
Andreea Pavel
Connor Sheehan
Margareta Eliza Balazs
Bogdan Tara
arthur.iakab
Ted Mielczarek
Masatoshi Kimura
Dorel Luca
Robert Bartlensky
Jim Chen
Chris Peterson
Henrik Skupin
Daniel Varga
Cosmin Sabou
Andreas Tolfsen
Tom Ritter
Robert Longson
Sebastian Hengst
ui.manish
Myk Melez