We can't use taskcluster 5.0.0 yet, because taskcluster-proxy does not
support new-style URLs.
Differential Revision: https://phabricator.services.mozilla.com/D10146
--HG--
extra : moz-landing-system : lando
So we can generate generic enum by cbindgen (for TimingFunction).
Depends on D9845
Differential Revision: https://phabricator.services.mozilla.com/D9312
--HG--
extra : moz-landing-system : lando
The function is called a lot for the same paths and is rather costly, so
cache the results for each path.
Depends on D9758
Differential Revision: https://phabricator.services.mozilla.com/D9759
--HG--
extra : moz-landing-system : lando
SourceLocation that are passed to inThirdPartyPath might be macro
expansion locations, for which SourceManager.getFilename returns the
path of the directory containing the source, rather than of the
expansion location.
Furthermore, the paths getFileName returns are not canonical, and can
contain e.g. `..`.
Differential Revision: https://phabricator.services.mozilla.com/D9758
--HG--
extra : moz-landing-system : lando
This is the equivalent of the rustc-workspace-hack used by the rust build to
ensure cargo and RLS see the same set of features for dependencies so that
these dependencies may be reused by invocations of cargo for these two
projects. The trivial crate added specifies the union of the set of
features activated for a particular crate for each time it appears in the
dependency tree so that cargo will understand these dependencies to be
re-usable across cargo implementations. This eliminates re-building jsrust
and some of its dependencies twice, and reduces the number of crates compiled
in the tree by about 90 in testing on linux.
Differential Revision: https://phabricator.services.mozilla.com/D9041
This is some sort of followup to bug 1423813, providing a minimalistic
way to undo elfhack when the elfhack sections are in separate segments,
which has been the case since bug 1385783 but didn't cause problems
on Android builds until bug 1423822.
Depends on D9622
Differential Revision: https://phabricator.services.mozilla.com/D9623
--HG--
extra : moz-landing-system : lando
This effectively backs out bug 822584, which worked around a similar
problem to what we are facing with Android xpcshell, being that the
crash reporter doesn't handle the address space "fragmentation" induced
by elfhack. The work around worked, at the expense of some added
complexity.
It was used for B2G only, and has effectively been unused since B2G was
retired.
Differential Revision: https://phabricator.services.mozilla.com/D9622
--HG--
extra : moz-landing-system : lando
This is the equivalent of the rustc-workspace-hack used by the rust build to
ensure cargo and RLS see the same set of features for dependencies so that
these dependencies may be reused by invocations of cargo for these two
projects. The trivial crate added specifies the union of the set of
features activated for a particular crate for each time it appears in the
dependency tree so that cargo will understand these dependencies to be
re-usable across cargo implementations. This eliminates re-building jsrust
and some of its dependencies twice, and reduces the number of crates compiled
in the tree by about 90 in testing on linux.
Differential Revision: https://phabricator.services.mozilla.com/D9041
--HG--
extra : moz-landing-system : lando
The cctools-port linker links against libraries from clang (for LTO),
which have different SONAMEs depending on the clang version. Which means
the linker needs to be used along the same version of clang it was built
against. Thus we also make it depend on linux64-clang-7.
But changing the dependency is not enough, cf. bug 1471905, so also
touch its build script, which it turns out, we need to do anyways
because llvm-dsymutil was renamed to dsymutil.
Relatedly, all toolchains that are built using cctools-port need to use
linux64-clang-7 too.
Building compiler-rt 7 with the OSX 10.11 SDK fails because of some
newer APIs being used in compiler-rt for xray, but this is not a feature
we use, so disable that.
Differential Revision: https://phabricator.services.mozilla.com/D6766
The desired outcome of this change is that we'll set
-Wl,--version-script based on linker kind and not on the output of
$LINKER -v.
This is a cheap way to address a simple problem that has a complicated
ideal solution. The underlying issue is that in some situations, when
targeting Android, a macOS system ld is interrogated to determine if
a cross-compiling linker "is GNU ld" and a particular linker feature
is set in that situation. The macOS system ld doesn't pass the "is
GNU ld" test, and the linker feature isn't set; that causes link
failures, even though the actual linker has nothing to do with the
system ld.
The ideal solution is to test for linker capabilities dynamically. We
do a lot of that in old-configure.in, and we don't do any of that in
toolchain.configure. Rather than start testing in
toolchain.configure, we hard-code: a cheap solution to the immediate
problem.
MinGW suffers somewhat from the opposite problem: the linker "is GNU
ld" (compatible), but the linker checks don't happen at all. We hard-code
for MinGW based on the C compiler instead.
Differential Revision: https://phabricator.services.mozilla.com/D8471
--HG--
extra : moz-landing-system : lando
This patch also changes how pdbs for the ASAN job are copied:
we relax restrictions so that pdbs if present) are always copied out
and add an environment variable MOZ_COPY_PDBS to indicate when we
want to produce pdbs for copying.
The desired outcome of this change is that we'll set
`-Wl,--version-script` based on linker kind and not on the output of
`$LINKER -v`.
This is a cheap way to address a simple problem that has a complicated
ideal solution. The underlying issue is that in some situations, when
targeting Android, a macOS system `ld` is interrogated to determine if
a cross-compiling linker "is GNU ld" and a particular linker feature
is set in that situation. The macOS system `ld` doesn't pass the "is
GNU ld" test, and the linker feature isn't set; that causes link
failures, even though the actual linker has nothing to do with the
system `ld`.
The ideal solution is to test for linker capabilities dynamically. We
do a lot of that in old-configure.in, and we don't do any of that in
toolchain.configure. Rather than start testing in
toolchain.configure, we hard-code: a cheap solution to the immediate
problem.
Differential Revision: https://phabricator.services.mozilla.com/D8471
--HG--
extra : moz-landing-system : lando
While we do have some uses of @depends-function comparison in some
templaces, related to host/target, we ought to be using `is` comparisons
rather than `==` anyways, so we switch those, and prevent other kinds of
comparisons being used at all.
This unveils the one noted in
https://phabricator.services.mozilla.com/D7713?id=21357#inline-30414
(and surprisingly only that one), that we remove entirely since it was
doing nothing in practice. Bug 1492305 will have to add it back in a
proper form.
Differential Revision: https://phabricator.services.mozilla.com/D8501
--HG--
extra : moz-landing-system : lando
This commit adds some documentation about build telemetry
to the build system documentation. This doc should be used
as a reference during opt-in to build system telemetry,
and includes paths to the relevant files for users to audit
independently. Documentation regarding the schema types and
object properties was generated by the `jsonschema2rst` tool,
available on PyPI.
Differential Revision: https://phabricator.services.mozilla.com/D8030
--HG--
extra : moz-landing-system : lando
If scandir is already present on the system the attempt to import the
c helper library will currently find the c helper from the system
install which may well be an outdated verion, so causing mach to
break. To solve this this patch does two things:
* Stops importing scandir in files that are run unconditionally when
invoking mach. This is generally considered good for performance
reasons.
* Installs the vendored scandir into the virtualenv for `mach lint`
rather than trying to import it directly from the source tree and so
not getting the c helper library.
Differential Revision: https://phabricator.services.mozilla.com/D8379
--HG--
extra : moz-landing-system : lando
This also requires the 64-bits rust compiler and some build system
tweaks.
And since we make the 32-bits builds cross-compiles on CI, we also need
to adjust the MSVC build mozconfigs such that the host compiler points
to the right MSVC cl. Likewise, the DIA SDK is used for host things, so
use the 64-bits version or it.
Differential Revision: https://phabricator.services.mozilla.com/D7845
--HG--
extra : moz-landing-system : lando
Running rustc with an open fd 4 sometimes causes it to fail with an obscure
error when it tries to connect to a job server on that fd. Closing the fd
solves the problem.
close_fds does what we need, and is the default value in Python 3, but not
in Python 2.
Differential Revision: https://phabricator.services.mozilla.com/D8167
--HG--
extra : rebase_source : d94447019e3e698d7cf2c293b0a9b99769cf316a
This patch adds two new actions to virtualenv_packages.txt processing:
windows and !windows. The former processes the rest of the action only on
Windows, and the latter processes it only on non-Windows.
These new features are used in virtualenv_packages.txt to use the
path to the unpacked Windows psutil wheel when on Windows, and build psutil
from source and use that path on other platforms.
This fixes the long-standing problem of not having psutil available on most
Windows systems (since they don't have the right set of Visual C++ build tools).
Differential Revision: https://phabricator.services.mozilla.com/D3437
--HG--
extra : moz-landing-system : lando
Scandir is a faster implementation of os.listdir since it caches file metadata
as it works. Using listdir and then calling things like os.path.isfile after
the fact will result in multiple system calls. Whereas with scandir, there will
only be a single system call per file.
Scandir is part of the stdlib in Python 3.6+, so the following can be used for
Python 2/3 compatible code:
try
from os import scandir, walk
except ImportError:
from scandir import scandir, walk
Differential Revision: https://phabricator.services.mozilla.com/D7738
--HG--
extra : moz-landing-system : lando
The incorrect LEGACY_BUILD_ID will be fixed in a subsequent changeset.
We must add https://www.mozilla.org/ to server-locations.txt and regenerate the mochitest certificates [1] because the new navigator.buildID test pretends to load content from https://www.mozilla.org/.
[1] https://searchfox.org/mozilla-central/source/build/pgo/certs/README
Differential Revision: https://phabricator.services.mozilla.com/D7982
--HG--
rename : dom/tests/mochitest/bugs/test_bug351601.html => dom/tests/mochitest/bugs/test_navigator_buildID.html
extra : rebase_source : 1deb142930f1a7a570cf719c4cb2bed8adfeabe2
extra : source : 408bff32f9623513a271cdf043d11ba6d1318e03
The cctools-port linker links against libraries from clang (for LTO),
which have different SONAMEs depending on the clang version. Which means
the linker needs to be used along the same version of clang it was built
against. Thus we also make it depend on linux64-clang-7.
But changing the dependency is not enough, cf. bug 1471905, so also
touch its build script, which it turns out, we need to do anyways
because llvm-dsymutil was renamed to dsymutil.
Relatedly, all toolchains that are built using cctools-port need to use
linux64-clang-7 too.
Building compiler-rt 7 with the OSX 10.11 SDK fails because of some
newer APIs being used in compiler-rt for xray, but this is not a feature
we use, so disable that.
Differential Revision: https://phabricator.services.mozilla.com/D6766
Bug 1282866 removed the only configuration where it did something.
Since then, using it only leads to a configure error.
Differential Revision: https://phabricator.services.mozilla.com/D7691
--HG--
extra : moz-landing-system : lando
We need to sign parts of the contents of the archives, so the mar's that we
ship get built as part of the repackage task. Thus, there is no reason to also
create and upload as part of the build, just to throw them away.
Differential Revision: https://phabricator.services.mozilla.com/D6213
--HG--
extra : moz-landing-system : lando
The cctools-port linker links against libraries from clang (for LTO),
which have different SONAMEs depending on the clang version. Which means
the linker needs to be used along the same version of clang it was built
against. Thus we also make it depend on linux64-clang-7.
But changing the dependency is not enough, cf. bug 1471905, so also
touch its build script, which it turns out, we need to do anyways
because llvm-dsymutil was renamed to dsymutil.
Relatedly, all toolchains that are built using cctools-port need to use
linux64-clang-7 too.
Building compiler-rt 7 with the OSX 10.11 SDK fails because of some
newer APIs being used in compiler-rt for xray, but this is not a feature
we use, so disable that.
Differential Revision: https://phabricator.services.mozilla.com/D6766
--HG--
rename : build/build-clang/clang-6-macosx64.json => build/build-clang/clang-7-macosx64.json
rename : taskcluster/scripts/misc/build-clang-6-linux-macosx-cross.sh => taskcluster/scripts/misc/build-clang-7-linux-macosx-cross.sh
extra : moz-landing-system : lando
This incorporates some fixes for ccache, icecream, and a better display
output (decreasing count instead of an increasing count). It may also
let us experiment with incremental rust support, though I think that
feature still needs some more work.
MozReview-Commit-ID: 7zPGiL9Ob6N
Differential Revision: https://phabricator.services.mozilla.com/D7152
--HG--
extra : moz-landing-system : lando
Some build flags are being passed by the build system: they're passed in a text
file called extra-bindgen-flags.in that's filled at configure time.
Other flags have to be inferred from the current target/host combination, in
Cranelift's build script directly. This is mostly cargo-culted from the
ServoBindings.toml file, and should probably be merged in the build system at
some point.
Some Windows-specific adjustments were needed to provide access to libclang for
bindgen support, by adding clang-cl to the plain Spidermonkey Windows builds.
--HG--
extra : rebase_source : 0bda40b1d1eb38c2657593f094c951013711d00a
extra : histedit_source : aad930a5f9099e299d385ae4de2deb81aed9b6d5
The ASan runtime wasn't designed to be unloaded, so pin it in memory.
Differential Revision: https://phabricator.services.mozilla.com/D6943
--HG--
extra : rebase_source : 843fc91b3150f3b264e321ab89bf723a01fc8b3c
With libLLVM being a shared library exporting many symbols, all internal
calls using those symbols default to go through the PLT, which is
unnecessary (and costly) overhead. Using -Bsymbolic-functions makes
internal calls go directly to the right place without going through the
PLT.
Differential Revision: https://phabricator.services.mozilla.com/D7029
With libLLVM being a shared library exporting many symbols, all internal
calls using those symbols default to go through the PLT, which is
unnecessary (and costly) overhead. Using -Bsymbolic-functions makes
internal calls go directly to the right place without going through the
PLT.
Differential Revision: https://phabricator.services.mozilla.com/D7029
This extracts the current logic for finding nodejs into its own module in mozbuild. Configure and ESLint then use it.
For ESLint, this will change the first location it looks for nodejs to be the .mozbuild directory.
Differential Revision: https://phabricator.services.mozilla.com/D6430
--HG--
extra : moz-landing-system : lando
This change prevents javascript:-URLs from being passed at the command
line. This restriction was already applied to every URL but the first.
Even the first URL did not result in any visible effect when Firefox is
started. Yet somehow the PGO profiler script managed to rely on it.
This commit ensures that "javascript:" URLs are not activated regardless of
position in the command line, and switches to a data:-URL for the PGO script to
achieve the (previously) desired effect of quitting the browser on startup.
Depends on D4929
Differential Revision: https://phabricator.services.mozilla.com/D5958
--HG--
extra : moz-landing-system : lando
This patch gets us building an updater binary that always embeds the dep certificates (instead of release or nightly), and builds a new tests package that includes it.
This was originally D5900, but that was backed out due to busting artifact builds. I've fixed that by removing the Makefile that Ted pointed out is unnecessary.
Differential Revision: https://phabricator.services.mozilla.com/D6418
--HG--
extra : moz-landing-system : lando
ParmVarDecl being a subclass of VarDecl, using two matchers then caused
ScopeChecker::check to be called twice for ParmVarDecl nodes, once for
each match. But the code in ScopeCheck::check is written with the
assumption that it's called only once for such nodes.
Somehow, this didn't cause problems with clang up to version 6, but
makes the plugin spuriously warn about already_AddRefed not being used
as temporaries when used as argument in function declarations, with
clang 7.
Differential Revision: https://phabricator.services.mozilla.com/D6360
--HG--
extra : moz-landing-system : lando
Doing so changes the size of the compressed toolchain archive from ~280M
to ~120M, and the decompressed size from ~1500M to ~675M. This will
reduce the overhead of decompression during builds.
As we ship llvm-symbolizer as part of ASan builds, we do need it to
still statically link against LLVM, which we do with a small patch.
With LLVM as a shared library, libLTO, which is used by cctools-port for
the linker, is dynamically linked to LLVM, and the cctools-port
configure script fails to link against libLTO. So we add a -rpath-link
to make it find the LLVM library. This happens to force a rebuild of
cctools-port, but for future cases where we might need a rebuild because
of some clang changes, we add a comment to ease the process, and avoid
a newer cctools-port taking the cache spot of an older one.
Ideally, mac cctools-port would need something similar, but it needs a
mac libLTO.dylib, which is not there anyways (and the mac cctools-port
thus already didn't support LTO).
Also, with LLVM built as a shared library, all its symbols are exported
with a LLVM_x.y version. Combined with -static-libstdc++ that is used
during the clang build, this causes problems (see
https://bugzilla.mozilla.org/show_bug.cgi?id=1492037#c7). But it turns
out things have evolved since -static-libstdc++ has been added to the
clang build script, and things work without now, so remove it (as well
as -static-libgcc).
Differential Revision: https://phabricator.services.mozilla.com/D6117
Doing so changes the size of the compressed toolchain archive from ~280M
to ~120M, and the decompressed size from ~1500M to ~675M. This will
reduce the overhead of decompression during builds.
As we ship llvm-symbolizer as part of ASan builds, we do need it to
still statically link against LLVM, which we do with a small patch.
With LLVM as a shared library, libLTO, which is used by cctools-port for
the linker, is dynamically linked to LLVM, and the cctools-port
configure script fails to link against libLTO. So we add a -rpath-link
to make it find the LLVM library. This happens to force a rebuild of
cctools-port, but for future cases where we might need a rebuild because
of some clang changes, we add a comment to ease the process, and avoid
a newer cctools-port taking the cache spot of an older one.
Ideally, mac cctools-port would need something similar, but it needs a
mac libLTO.dylib, which is not there anyways (and the mac cctools-port
thus already didn't support LTO).
Also, with LLVM built as a shared library, all its symbols are exported
with a LLVM_x.y version. Combined with -static-libstdc++ that is used
during the clang build, this causes problems (see
https://bugzilla.mozilla.org/show_bug.cgi?id=1492037#c7). But it turns
out things have evolved since -static-libstdc++ has been added to the
clang build script, and things work without now, so remove it (as well
as -static-libgcc).
Differential Revision: https://phabricator.services.mozilla.com/D6117
This commit updates submit_telemetry_data.py to send data
to the Telemetry pipeline. The script assumes the presence
of a "telemetry" directory within the statedir, and an
"outgoing" directory within the "telemetry" directory (otherwise
there is no data to submit). The script will create a
"submitted" directory and "telemetry.log" file if absent,
making the assumption that this is the first build telemetry
submission for that user. UUID values for submitted data points
are seeded from the filename, without the ".json" suffix.
Differential Revision: https://phabricator.services.mozilla.com/D5687
--HG--
extra : moz-landing-system : lando
This patch gets us building an updater binary that always embeds the dep certificates (instead of release or nightly), and builds a new tests package that includes it.
I had a lot of trouble getting the test package generated correctly due to the fact that things in "_tests" won't be included for test packages that aren't "common". My fix for that isn't ideal - I'm open to something better.
Differential Revision: https://phabricator.services.mozilla.com/D5900
--HG--
extra : moz-landing-system : lando
Doing so changes the size of the compressed toolchain archive from ~280M
to ~120M, and the decompressed size from ~1500M to ~675M. This will
reduce the overhead of decompression during builds.
As we ship llvm-symbolizer as part of ASan builds, we do need it to
still statically link against LLVM, which we do with a small patch.
CCTools-port needs to be rebuilt in some cases of clang changes, which
this one of, so touch the script so that it happens.
Differential Revision: https://phabricator.services.mozilla.com/D6117
BUILD_TOOLS was only ever used for things that another variable provides
equally well. Removing BUILD_TOOLS means that we can remove win_srcdir
and WIN_TOP_SRC as well.
All but one of the current uses of DEFFILE use `SRCDIR + '/file.def'` to
get a srcdir-relative path anyway, and the other one wants an
objdir-relative path, so using Path makes everything clearer.
This makes it more straightforward to translate the paths for the WSL
build.
- We forcefully remove annotations from the AST so that they don't end
up impacting codegen.
- We change the API such that we use identifiers instead of strings,
reducing the chances of typo errors.
Differential Revision: https://phabricator.services.mozilla.com/D5493
--HG--
extra : moz-landing-system : lando
We were already always enabling it on builds with frame pointers on
Android. We now extend this to builds without frame pointers, helping
with client-side stacktracing for crashes.
It's also the default on many platforms, such as x86-64, whether frame
pointer are enabled or not and on x86 with GCC. Somehow, it's not the
case for x86 with clang, so this makes clang builds the same as GCC
builds in that regard.
On ARM, we were always omitting frame pointers because of a GCC bug, but
we've not been using GCC for Android builds for a long time (and the GCC
bug was actually fixed in 4.7 and backported to 4.6, making it fixed in
all versions of GCC we support).
Differential Revision: https://phabricator.services.mozilla.com/D6110
--HG--
extra : moz-landing-system : lando
This patch makes it so we write the sccache log directly to the artifact
directory, so that it will be uploaded even if the build fails. It also makes
the log slightly more verbose. Both of these should help with diagnosing
sccache failures in CI.
The sccache log will no longer be explicitly gzip compressed, but some
Taskcluster client implementations will store it as gzip compressed.
Differential Revision: https://phabricator.services.mozilla.com/D6187
--HG--
extra : moz-landing-system : lando
When checking whether the new relocations sizes are going to be a win, we
need to account for the fact that there are non-elfhacked relocation
remaining.
Differential Revision: https://phabricator.services.mozilla.com/D5837
--HG--
extra : moz-landing-system : lando
Now that we ship builds using clang on all platforms, pick it during
configure. It is still possible to opt-in to building other compilers by
setting CC/CXX (or even only CC) to the desired compiler.
Depends on D5829
Differential Revision: https://phabricator.services.mozilla.com/D5637
--HG--
extra : moz-landing-system : lando
Before this change, we'd derive a host compiler and handle things as if
HOST_CC/HOST_CXX had been passed. With this change, we change the list
of compilers that are tested with check_prog instead.
Depends on D5828
Differential Revision: https://phabricator.services.mozilla.com/D5829
--HG--
extra : moz-landing-system : lando
Currently, when building for Android x86-64 on Linux x86-64, we drop the
'--target' flag, which causes the build to fail. This patch adds a check
for OS mismatch, so we keep the '--target' flag in this situation.
Differential Revision: https://phabricator.services.mozilla.com/D4483
Ciure Andrei
Tom Prince
Emilio Cobos Álvarez
Margareta Eliza Balazs
Boris Chiou
Mike Hommey
David Major
Chris Manchester
Csoregi Natalia
Jeff Gilbert
Kartikaya Gupta
Tom Ritter
Nick Alexander
Chris Peterson
Dorel Luca
Benjamin Bouvier
Cosmin Sabou
Connor Sheehan
Narcis Beleuzu
James Graham
Tiberius Oros
Kris Maglione
Dave Hunt
Ted Mielczarek
Andrew Halberstadt
Sylvestre Ledru
Brindusan Cristian
Noemi Erli
Guillaume GARDET
shindli
Mike Shal
Gurzau Raul
Nathan Froyd
Coroiu Cristina
Mark Banner
Andi-Bogdan Postelnicu
Rob Wu
Ben Hearsum
Thomas P.
Andreea Pavel
Bogdan Tara
arthur.iakab
Masatoshi Kimura
Robert Bartlensky
Jim Chen