The GNU_AS check in old-configure depended on running with the value
of $AS before it gets reset to just be the C compiler, which breaks when
we move setting AS into moz.configure.
This patch moves the GNU_AS check to toolchain.configure and changes it
so that it works when the assembler is the C compiler. We do have to
fix things slightly for clang, because the previous check was
succeeding, but not because of clang: it was detecting the presence of
"GNU" in the output for GNU ld/gold and a message about the GNU GPL.
This is a fairly straightforward port of the AS tool checks from old-configure
to toolchain.configure. AS is a little quirky in that we currently do a
normal-looking check for it, but then override that value to be the C compiler
for non-Windows builds, and ml[64]/armasm64 for Windows builds.
After migrating those checks, the only things left in the MOZ_DEFAULT_COMPILER
macro in compiler-opts.m4 were some unused bits, so I removed them:
* Setting of CPP/CXXCPP, which are set in toolchain.configure now
* Setting HOST_LDFLAGS to empty, which doesn't seem particularly useful.
There was also a quirky old test that the assembler was ml[64] when js-ctypes
is enabled that I removed, I don't think it provides any value since this
patch will ensure that we're using the right assembler for Windows builds.
This setup seems to work well enough to enable me to link
HOST_SIMPLE_PROGRAMS with an AArch64-cross setup. Necessary library
paths are passed to the linker via -LIBPATH and HOST_LDFLAGS rather than
letting MSVC fish them out of the environment. The change to
HOST_SIMPLE_PROGRAMS to pass HOST_LDFLAGS was necessary for this to
work, in addition to the HOST_LINKER changes.
-Wtautological-overlap-compare is an opt-in warning added in clang 3.5. It warns about overlapping comparisons that are always true or false, such as:
if (x > 4 || x < 10) {} // warning! always true
int b = x < 2 && x > 5; // warning! always false
return x > 4 || x < 10; // warning! always true
https://clang.llvm.org/docs/DiagnosticsReference.html#wtautological-overlap-compare
There are currently no -Wtautological-overlap-compare warnings in mozilla-central.
Differential Revision: https://phabricator.services.mozilla.com/D3477
--HG--
extra : rebase_source : e6aa211a338633e5fb7507b3a8f341709d72d6c9
This opt-in warning catches bugs where a constructor modifies a constructor parameter that shadows member variable name. The code probably intended to change the member variable value, not the paramter. There are currently no -Wshadow-field-in-constructor warnings in mozilla-central.
https://clang.llvm.org/docs/DiagnosticsReference.html#wshadow-field-in-constructor-modified
Differential Revision: https://phabricator.services.mozilla.com/D3822
--HG--
extra : rebase_source : 604ff90e036d536955db41351ee9ee97e8424d92
extra : intermediate-source : 712f538afc160f4efae2383130727edbd1dd70f6
extra : source : 300fff9a58515c7cac0647afc9ccc0e69bc8c5cb
-Wfloat-overflow-conversion detects when a constant floating point value is converted to an integer type and will overflow the target type.
https://clang.llvm.org/docs/DiagnosticsReference.html#wfloat-overflow-conversion
-Wfloat-zero-conversion detects when a non-zero floating point value is converted to a zero integer value.
https://clang.llvm.org/docs/DiagnosticsReference.html#wfloat-zero-conversion
There are currently no -Wfloat-overlap-conversion warnings in mozilla-central. There is one -Wfloat-zero-conversion warning in a webrtc test. It doesn't block enabling this check because the webrtc tests are not compiled with warnings-as-errors.
media/webrtc/trunk/webrtc/modules/audio_coding/audio_network_adaptor/frame_length_controller_unittest.cc:255:54 [-Wfloat-zero-conversion] implicit conversion from 'const float' to 'int' changes non-zero value from 0.045000002 to 0
We can't enable all -Wfloat-conversion warnings (for any implicit conversion of a floating-point number into an integer) because there are currently over 1400 warnings. I spot checked a few of these -Wfloat-conversion warnings. I didn't find any obvious bugs, but there is some suspicious code, such as implicit conversions of floats to bools.
Differential Revision: https://phabricator.services.mozilla.com/D3476
--HG--
extra : rebase_source : 589be9e593749c7000b1f89ca155e4f95a487f8f
Warn about C++ constructs whose meaning change in C++2a.
https://clang.llvm.org/docs/DiagnosticsReference.html#wc-2a-compat
So far the only -Wc++2a-compat check that I know of is for valid pre-C++2a code that inadvertently parses as C++2a's new <=> "spaceship" comparison operator. `f<&A::operator<=>();` is an example of a warning reported for a real project on GitHub. That code can be rewritten as `f< &operator<= >();`.
There are currently no -Wc++2a-compat warnings in mozilla-central.
Differential Revision: https://phabricator.services.mozilla.com/D3478
--HG--
extra : rebase_source : 4ec8147ad3fe2d1ef7c543d100c660a000385786
Our version check is removed as well, as the current required rustc version for
Firefox includes all features required to build with tup.
Differential Revision: https://phabricator.services.mozilla.com/D3847
--HG--
extra : moz-landing-system : lando
When valgrind prints out backtraces, it prints raw addresses and symbol
names, but that doesn't help find the exact code that caused the errors,
because we don't know where the libraries are loaded.
With --sym-offsets=yes, it adds the offset from the symbol, which allows
to find the relevant code in the binary.
For some reason, clang 6 crashes with a stack overflow on PGO + LTO on
Linux 64 bits. Clang 7 doesn't, but has other problems.
After some bisecting, I found the following:
- r322684 is the first revision that is broken on the release_60 branch.
- that revision is a cherry pick of r322313 from trunk, which is
similarly broken.
- trunk was fixed by r322325, which, funnily enough, predates when
r322313 was cherry-picked.
While the change from r322325 is relatively large, mixing multiple
different changes in a single commit, there also haven't been
significant changes to the same file on trunk since (one macro name
change, one documentation change, and a change related to debug info),
which would tend to indicate the change is not going to break anything,
or at least not more than upgrading to clang 7 would.
The exact part that fixes the issue could probably be found in this
large commit, but I didn't feel like digging into it further considering
the above.
In order to support operator==() for tagged enum, we have to bump the version to
0.6.2.
Differential Revision: https://phabricator.services.mozilla.com/D3932
--HG--
extra : moz-landing-system : lando
yasm doesn't support aarch64, and trying to use GNU as with an MSVC
build seems like sadness waiting to happen. Instead, we'll generate our
own assembly file that armasm64 will accept.
This change switches most CI builds to clang, with a few exceptions:
- valgrind builds, until bug 1481670 is figured out.
- PGO and nightly builds, until that's fully tested.
- coverage builds, per bug 1471339 comment 17.
- base toolchain builds, to keep some builds on GCC even when we're
fully switched to clang.
- any build that doesn't use build/unix/mozconfig.linux (e.g. probably
all those driven by autospider.py, maybe others).
This change adds a voluptuous schema for build system telemetry, replacing
the existing json schema file. Using voluptuous will make it easier to work
with the schema from Python code in the build system. A future commit will
use a Python module to provide a mach command to convert the voluptuous
schema to json schema format for consumption by other systems.
Differential Revision: https://phabricator.services.mozilla.com/D2840
--HG--
extra : rebase_source : 067995385334d1dbc123f2db4245ef4e69d076c3
In some rare cases, it is possible for one of the eh_frame sections'
original address to be larger than the address of the injected code
section, which is added before the first executable section. Namely,
this happens in the rare case where that eh_frame section is smaller
than the injected code section, and is adjacent to the first executable
section. We obviously want to move the eh_frame sections in that case,
since one of them is in the way.
We're currently using NDK r15c, which is rather old, and happens to come
with a buggy gold linker. Let's use a more recent NDK, with a fixed
linker.
Unfortunately, we're currently at NDK API level 9, which the newer NDK
doesn't provide for x86 anymore. But that corresponds to Gingerbread
(2.3), which we've long stopped supporting. On the SDK side, we already
dropped support of versions before Jelly Bean, so we can do the same on
the NDK side. That corresponds to API level 16. So let's just use that
as a baseline.
Another change in the newer NDK is that the target-name changed from
i386-linux-android to i686-linux-android, so adjust for that in the
android x86 mozconfigs.
We're currently using NDK r15c, which is rather old, and happens to come
with a buggy gold linker. Let's use a more recent NDK, with a fixed
linker.
Unfortunately, we're currently at NDK API level 9, which the newer NDK
doesn't provide for x86 anymore. But that corresponds to Gingerbread
(2.3), which we've long stopped supporting. On the SDK side, we already
dropped support of versions before Jelly Bean, so we can do the same on
the NDK side. That corresponds to API level 16. So let's just use that
as a baseline.
Another change in the newer NDK is that the target-name changed from
i386-linux-android to i686-linux-android, so adjust for that in the
android x86 mozconfigs.
This change switches most CI builds to clang, with a few exceptions:
- valgrind builds, until bug 1481670 is figured out.
- PGO and nightly builds, until that's fully tested.
- coverage builds, per bug 1471339 comment 17.
- base toolchain builds, to keep some builds on GCC even when we're
fully switched to clang.
- any build that doesn't use build/unix/mozconfig.linux (e.g. probably
all those driven by autospider.py, maybe others).
We already copy the 64-bits libraries, but don't copy the 32-bits
libraries, which prevents building for linux32 by default.
Incidentally, this also makes the clang build system build the 32-bits
compiler-rt libraries, allowing e.g. 32-bits PGO.
Bug 1423822 moved the injected code section before the .text section.
When linking with lld, the text section is usually page aligned, and
starting a PT_LOAD. We inject code at the beginning of the PT_LOAD,
which means the PT_LOAD is going to be extended at least a page
downwards. And it means the preceding PT_LOAD can't finish in that same
page, so the overhead of the injected code is needs to account for the
page alignment.
If the .eh_frame_hdr and .eh_frame sections are not between the elfhack
relocation and elfhack code sections, it's not going to change anything
to try to move it, so don't even try.
While here, adjust the adjacency test to error out when the section name
doesn't match, and account for the fact that the eh_frame_hdr section
might appear after eh_frame.
--HG--
extra : rebase_source : 7d3525abe75b5a014b39ce0bd406e8f78592ec39
All CI builds for linux32 have been running on 64-bits hosts for a long
while now, and there are no 32-bits hosts left. We can remove the
special-casing that makes the linux32 builds behave differently
depending on the bit-width of the host.
Furthermore, configure has been able to deal with adding -m32 on its
own, so we don't need to add it to $CC/$CXX manually anymore.
We already copy the 64-bits libraries, but don't copy the 32-bits
libraries, which prevents building for linux32 by default.
Incidentally, this also makes the clang build system build the 32-bits
compiler-rt libraries, allowing e.g. 32-bits PGO.
We only enable libstdc++ compat on Android for the host parts, and
in practice, the target part has it not enabled at the moment because
libstdc++ can't be found. But making the clang toolchain capable of
building for x86 changes the deal for Android x86.
If the .eh_frame_hdr and .eh_frame sections are not between the elfhack
relocation and elfhack code sections, it's not going to change anything
to try to move it, so don't even try.
While here, adjust the adjacency test to error out when the section name
doesn't match.
--HG--
extra : rebase_source : 4b31712576fd3472bb94a2b9ab9542253f04cba8
Somehow, when building with LTO, clang can end up creating a eh_frame
section with only one, empty, entry (which just looks like a 4-bytes
long section full of 0x00).
--HG--
extra : rebase_source : 385c05c7e447fe1c4bc261b79c7d56138e268458
We need to add mappings for target.cpu to MSVC's arm64 name for
determining various paths, and we need to add an extra case to
get_vc_paths so the compiler can find all the necessary DLLs.
This patch adds JaCoCo as a dependency for the geckoview androidTest configurations, as well as
the `mach android archive-geckoview-coverage-artifacts` command, and the `--enable-java-coverage`
mozconfig flag.
MozReview-Commit-ID: 36jNAzK44g3
--HG--
extra : rebase_source : 9edc37913a3929ad045270c601c77791d122e363
For the same reason test-array.c and test-ctors.c need to be built
explicitly without LTO.
--HG--
extra : rebase_source : d037ef7cf1dd2d278c2918dbfee5b4f4c213e408
When linking with ld.bfd or gold, this changes the PT_LOAD in which the
elfhack code section ends up, making it go in the same one as .init, .text,
etc. rather than .rel.*. When linking with lld, this completely
avoids doing a PT_LOAD split, because lld already splits .rel.* and
.text.
--HG--
extra : rebase_source : 1f69b8f4b48b055892ea24eaa6226859cc4ffd50
We treat segments overlapping as a fatal error, rather than a condition
to do nothing, because it happening is usually the result of some bad
assumptions on the input ELF, and we don't want to silently ignore
those.
However, there are cases where a setup /could/ lead to overlapping
segments, but would be skipped because elfhack wouldn't be a win
anyways. By checking segments overlap later, we allow those to not
hard fail.
--HG--
extra : rebase_source : deca2051722aeaa959c5e4dae06642908f6d843a
The current check makes assumption wrt what PT_LOAD the injected sections
end up in, and won't work with upcoming changes.
--HG--
extra : rebase_source : b7cfb65ea13c16f977fe523aaf9f39eafeb2cdce
Summary:
The plan is to also expose perfecthash.py from this module on the python path.
This also allows us to stop using explicit module loading to load make_dafsa.py.
make_dafsa.py was moved into tools/ to avoid any extra python files from
accidentally ending up on the python path.
Reviewers: froydnj!
Tags: #secure-revision
Bug #: 1479484
Differential Revision: https://phabricator.services.mozilla.com/D2614
--HG--
rename : xpcom/ds/make_dafsa.py => xpcom/ds/tools/make_dafsa.py
When compiling C/C++ sources via Rust build scripts there's no point in
enabling warnings-as-errors as all such code is third-party code.
MozReview-Commit-ID: 5pGH6w9ZE2I
--HG--
extra : rebase_source : ad36be6aed39d55055728f85b799adf5f0cf0f4b
https://github.com/rust-lang/rust/issues/52694 is a miscompilation I
found in rust when it uses system llvm 5 or 6, that was fixed 5 months
ago in the llvm rust bundles. This may or may not affect clang, but
considering it was also reported to upstream llvm independently of rust,
it's better to side with caution.
It doesn't affect 3.9, and bug 1478919 got rid of the last use of clang
5 (except for clang-tidy, but that's not used to compile).
The patches come from the llvm trunk from 5 months ago, so they're
already in our clang 7 snapshots.
Windows static analysis builds are still using an old trunk, but are
stuck on bug 1427808. They're "only" for static analysis, though.
--HG--
extra : rebase_source : f4fce69eb7c69b6245518a1bad37e04236c7075b
This was cargo-culted from the asan/tsan mozconfigs, but is not necessary
for builds without sanitizers.
--HG--
extra : rebase_source : 41bad4761f424410cb7a099ecaecce8a86becf59
JS, at least, doesn't need bindgen at this point. If JS does start
requiring bindgen under certain build configurations, we'll have to
figure out what to do about the bindgen dependency at that point:
require bindgen always, or just require bindgen for the
configuration(s).
Summary:
This adds a clangcl flag to gyp.mozbuild and then uses that to set the same
warnings flags for clang-cl as for clang.
Reviewers: dmajor
Tags: #secure-revision
Bug #: 1478000
Differential Revision: https://phabricator.services.mozilla.com/D2427
--HG--
extra : rebase_source : 6137d2f59e5504d448d4fb32a7e734756ff76afa
Summary:
We're currently returning a fake value on all of our automation builds. Might
as well not run the script at all.
Tags: #secure-revision
Bug #: 1371485
Differential Revision: https://phabricator.services.mozilla.com/D1803
--HG--
extra : rebase_source : 20717372c15547b35759ff69088aea1ca92b7198
This adds just enough host shared library support for this one use case,
but also takes shortcuts, because fully supporting host shared library
is a deep rabbit hole I'm not ready to take just to fix --enable-lto
--enable-clang-plugin on mac builds.
One downside is that one my machine the plugin now takes > 80s to build,
instead of 15s before, thanks to the lack of unified sources.
--HG--
extra : rebase_source : bf52a72a01d4e3eb77cf52b646b19734b9273075
Currently, --enable-lto just implies --enable-linker=lld, which
essentially only works on Linux, and assumes one can't do lto with
anything other than lld. Which is not true. As a matter of fact, even
ld.bfd can do lto, as long as the gold plugin for llvm is available,
or when doing lto with GCC instead of clang.
Anyways, to allow more lto setups, we adapt the --enable-linker setup
to:
- work on macOS, which it currently doesn't, and add support for the mac
linker (ld64), which, unfortunately, doesn't have a clean way to be
detected, so work around that.
- default to lld if lto is enable, no linker was explicitly given, the
compiler is clang *and* the build target is not macOS.
--HG--
extra : rebase_source : 1dab2ad6230d18e7f4285943e76352e23b987d4e
Because we have no linking configure test in python configure (yet), we
just make old-configure tests use LINKER_LDFLAGS, and make those flags
added to LDFLAGS by old-configure at the same time.
--HG--
extra : rebase_source : 80ab7c5021c7ddd1b53d58ef76cd4372a524d3cb
The intent is for the build system to soon require Node.js to build
Firefox. But we aren't ready to make Node.js a build requirement
just yet.
The goal of this commit is to implement configure detection for
Node.js so that we can a) work out detection bugs b) give people a
means to validate system compatibility *before* we throw the switch to
require Node.js.
This commit introduces configure logic for finding a Node.js
executable, resolving its version, and validating its suitability.
By default, if Node.js cannot be found or there is an error resolving
its version, we print some warning messages and move on.
If --enable-nodejs is used (not the default), errors are raised
if Node.js cannot be found or its version isn't suitable.
Once we require Node.js, the added code can likely be simplified.
When writing the code, I went out of my way to make failures as
non-fatal as possible. e.g. normally we'd say that failures to run
`node --version` would be fatal. I'm purposefully trying to not have
this configure check break anyone's environment, even if failure
occurs. Again, the goal is to introduce the configure checks first
in a non-fatal way such that we can debug failures so the flag day
transition is simpler.
Differential Revision: https://phabricator.services.mozilla.com/D1818
--HG--
extra : moz-landing-system : lando
Added `./mach python-safety`, distinct from python-test so it doesn't have
to be run on every CI job - its errors may not depend on the area the push has changed.
Added the python/safety directory to ensure a different Pipfile is used, avoiding
conflicts with python-test.
Differential Revision: https://phabricator.services.mozilla.com/D1825
--HG--
extra : moz-landing-system : lando
We're well overdue for an upgrade of the rust compiler requirements.
Now that we're building with 1.28 (albeit a beta, due to be bumped when
it's released), we can bump the requirement away from 1.24 which is now
old. 1.27 is too new, though, so settle for the older 1.26.
--HG--
extra : rebase_source : a17aa496bf3d4af4d1349d69a637c686c6817d0f
This will make sure that when running |mach python-test --python 3| locally,
we only run the tests that also run in CI with python 3 (and therefore pass
presumably).
MozReview-Commit-ID: 3OBr9yLSlSq
--HG--
extra : rebase_source : 456340d0ecdddf1078f2b5b4ebb1eddf3813b26a
The media/libpng/moz.build file overrides the C standard used via
-std=c89, per bug 1371266, which conflicts with the use of the
arm_neon.h header: compilation fails on the inline keyword, which didn't
exist in C89. We thus "bump" to the GNU89 standard, which is C89+GNU
extensions, including inline.
--HG--
extra : rebase_source : fe93a13e3bef8888e1874d2e94a6d8ef396aaf83
When I originally implemented bug 1458161, this is how it was done, but
it was suggested to use a configure-time check. This turned out to not
be great, because the rust compiler changes regularly, and we don't run
the configure tests when the version changes. When people upgraded their
rust compiler to 1.27, the code subsequently failed to build because the
features were still set for the previous version they had installed.
--HG--
extra : rebase_source : 1b5f7a02ad8495d68cd29289f7beea59b8912183
When a binary has a PT_GNU_RELRO segment, the elfhack injected code
uses mprotect to add the writable flag to relocated pages before
applying relocations, removing it afterwards. To do so, the elfhack
program uses the location and size of the PT_GNU_RELRO segment, and
adjusts it to be aligned according to the PT_LOAD alignment.
The problem here is that the PT_LOAD alignment doesn't necessarily match
the actual page alignment, and the resulting mprotect may end up not
covering the full extent of what the dynamic linker has protected
read-only according to the PT_GNU_RELRO segment. In turn, this can lead
to a crash on startup when trying to apply relocations to the still
read-only locations.
Practically speaking, this doesn't end up being a problem on x86, where
the PT_LOAD alignment is usually 4096, which happens to be the page
size, but on Debian armhf, it is 64k, while the run time page size can be
4k.
--HG--
extra : rebase_source : 5ac7356f685d87c1628727e6c84f7615409c57a5
We're well overdue for an upgrade of the rust compiler requirements.
Now that we're building with 1.28 (albeit a beta, due to be bumped when
it's released), we can bump the requirement away from 1.24 which is now
old. 1.27 is too new, though, so settle for the older 1.26.
--HG--
extra : rebase_source : c788ef4f7da9949b81df2f0577af6f6039ea63d8
When we're running using pipenv, we have more than one virtual environment. This means the current check to see if python matches the initial virtual environment gives a false positive when we're in a secondary virtual environment. This patch changes the condition to check if the current python path exists within the virtual environments root directory.
MozReview-Commit-ID: AAONwLWsigL
--HG--
extra : rebase_source : c0ac94448ee4545417b5116e58b51c6187cdb175
We used to do that before bug 1455767. Taking a step back, what we want
(and what this change implements) is to:
- allow to specify a linker to use explicitly
- check that using the corresponding linker flags makes us use that
linker
- if no linker is specified, and developer options are enabled, we want
to try to use gold if it's available, otherwise, we want to detect
what kind of linker the default linker is
- if --disable-gold is explicitly given, we don't want to automatically
use it.
--HG--
extra : rebase_source : 8d89c54bd5e555984d815beb8fdd3f5f75dae31e
Instead of clang 4, which they were the last to use, so remove the
clang 4 toolchain.
--HG--
extra : rebase_source : d03a083e9217aeb6c1d2c91decb978426f0e8d1a
The linux64-clang toolchain alias is currently clang 5. Switch it to
clang 6, but keep the spidermonkey tsan builds on clang 5 because of
bug 1467673.
The LLVM headers that come with clang 6 contain a DEBUG define that
conflicts with our DEBUG define and breaks the clang-plugin build,
so force unset ours.
--HG--
extra : rebase_source : aae88f1166108f003b06c022f14d5f4c61fc1ed9
Also work around https://bugs.llvm.org/show_bug.cgi?id=37746 by
explicitly handling ObjC interface variables separately. This actually
allows the searchfox macosx build to go much further than it used to (it
now fails during make package with apparently no output for rust code)
--HG--
extra : rebase_source : 354981ca9deebed5c60d3684f5c3abc554422393
We used to do that before bug 1455767. Taking a step back, what we want
(and what this change implements) is to:
- allow to specify a linker to use explicitly
- check that using the corresponding linker flags makes us use that
linker
- if no linker is specified, and developer options are enabled, we want
to try to use gold if it's available, otherwise, we want to detect
what kind of linker the default linker is
--HG--
extra : rebase_source : 655089f29c6403f6c31a977dcf5c6fd1e9941121
--enable-release not being passed means developer options are enabled,
which is generally speaking not desirable for builds meant to be
shipped. This is somewhat alleviated for Firefox by MOZILLA_OFFICIAL
implying --enable-release (as well as MOZ_AUTOMATION), but that doesn't
apply to e.g. standalone js builds (even some of the standalone js jobs
on our automation don't set MOZ_AUTOMATION for some reason).
A reasonable thing to do is just to default builds for release/beta
milestones to --enable-release, but still allow --disable-release
to enable the developer options.
--HG--
extra : rebase_source : 770d51b10a9cd17c63972435bb61eed10345ea71
Currently, many tasks fetch content from the Internets. A problem with
that is fetching from the Internets is unreliable: servers may have
outages or be slow; content may disappear or change out from under us.
The unreliability of 3rd party services poses a risk to Firefox CI.
If services aren't available, we could potentially not run some CI tasks.
In the worst case, we might not be able to release Firefox. That would
be bad. In fact, as I write this, gmplib.org has been unavailable for
~24 hours and Firefox CI is unable to retrieve the GMP source code.
As a result, building GCC toolchains is failing.
A solution to this is to make tasks more hermetic by depending on
fewer network services (which by definition aren't reliable over time
and therefore introduce instability).
This commit attempts to mitigate some external service dependencies
by introducing the *fetch* task kind.
The primary goal of the *fetch* kind is to obtain remote content and
re-expose it as a task artifact. By making external content available
as a cached task artifact, we allow dependent tasks to consume this
content without touching the service originally providing that
content, thus eliminating a run-time dependency and making tasks more
hermetic and reproducible over time.
We introduce a single "fetch-url" "using" flavor to define tasks that
fetch single URLs and then re-expose that URL as an artifact. Powering
this is a new, minimal "fetch" Docker image that contains a
"fetch-content" Python script that does the work for us.
We have added tasks to fetch source archives used to build the GCC
toolchains.
Fetching remote content and re-exposing it as an artifact is not
very useful by itself: the value is in having tasks use those
artifacts.
We introduce a taskgraph transform that allows tasks to define an
array of "fetches." Each entry corresponds to the name of a "fetch"
task kind. When present, the corresponding "fetch" task is added as a
dependency. And the task ID and artifact path from that "fetch" task
is added to the MOZ_FETCHES environment variable of the task depending
on it. Our "fetch-content" script has a "task-artifacts"
sub-command that tasks can execute to perform retrieval of all
artifacts listed in MOZ_FETCHES.
To prove all of this works, the code for fetching dependencies when
building GCC toolchains has been updated to use `fetch-content`. The
now-unused legacy code has been deleted.
This commit improves the reliability and efficiency of GCC toolchain
tasks. Dependencies now all come from task artifacts and should always
be available in the common case. In addition, `fetch-content` downloads
and extracts files concurrently. This makes it faster than the serial
application which we were previously using.
There are some things I don't like about this commit.
First, a new Docker image and Python script for downloading URLs feels
a bit heavyweight. The Docker image is definitely overkill as things
stand. I can eventually justify it because I want to implement support
for fetching and repackaging VCS repositories and for caching Debian
packages. These will require more packages than what I'm comfortable
installing on the base Debian image, therefore justifying a dedicated
image.
The `fetch-content static-url` sub-command could definitely be
implemented as a shell script. But Python is readily available and
is more pleasant to maintain than shell, so I wrote it in Python.
`fetch-content task-artifacts` is more advanced and writing it in
Python is more justified, IMO. FWIW, the script is Python 3 only,
which conveniently gives us access to `concurrent.futures`, which
facilitates concurrent download.
`fetch-content` also duplicates functionality found elsewhere.
generic-worker's task payload supports a "mounts" feature which
facilitates downloading remote content, including from a task
artifact. However, this feature doesn't exist on docker-worker.
So we have to implement downloading inside the task rather than
at the worker level. I concede that if all workers had generic-worker's
"mounts" feature and supported concurrent download, `fetch-content`
wouldn't need to exist.
`fetch-content` also duplicates functionality of
`mach artifact toolchain`. I probably could have used
`mach artifact toolchain` instead of writing
`fetch-content task-artifacts`. However, I didn't want to introduce
the requirement of a VCS checkout. `mach artifact toolchain` has its
origins in providing a feature to the build system. And "fetching
artifacts from tasks" is a more generic feature than that. I think
it should be implemented as a generic feature and not something that is
"toolchain" specific.
I think the best place for a generic "fetch content" feature is in
the worker, where content can be defined in the task payload. But as
explained above, that feature isn't universally available. The next
best place is probably run-task. run-task already performs generic,
very-early task preparation steps, such as performing a VCS checkout.
I would like to fold `fetch-content` into run-task and make it all
driven by environment variables. But run-task is currently Python 2
and achieving concurrency would involve a bit of programming (or
adding package dependencies). I may very well port run-task to Python
3 and then fold fetch-content into it. Or maybe we leave
`fetch-content` as a standalone script.
MozReview-Commit-ID: AGuTcwNcNJR
--HG--
extra : source : 0b941cbdca76fb2fbb98dc5bbc1a0237c69954d0
extra : histedit_source : a3e43bdd8a9a58550bef02fec3be832ca304ea93
After this change, we consistently import GPG keys from files in
the GCC build scripts.
MozReview-Commit-ID: BcyvCQoGbMS
--HG--
extra : source : 5fce34a460b51e45ac280a9f0cb8bad896fbcff1
extra : histedit_source : 01621ea8111315c251a9493a11efca72c2ba3c7d
Currently, many tasks fetch content from the Internets. A problem with
that is fetching from the Internets is unreliable: servers may have
outages or be slow; content may disappear or change out from under us.
The unreliability of 3rd party services poses a risk to Firefox CI.
If services aren't available, we could potentially not run some CI tasks.
In the worst case, we might not be able to release Firefox. That would
be bad. In fact, as I write this, gmplib.org has been unavailable for
~24 hours and Firefox CI is unable to retrieve the GMP source code.
As a result, building GCC toolchains is failing.
A solution to this is to make tasks more hermetic by depending on
fewer network services (which by definition aren't reliable over time
and therefore introduce instability).
This commit attempts to mitigate some external service dependencies
by introducing the *fetch* task kind.
The primary goal of the *fetch* kind is to obtain remote content and
re-expose it as a task artifact. By making external content available
as a cached task artifact, we allow dependent tasks to consume this
content without touching the service originally providing that
content, thus eliminating a run-time dependency and making tasks more
hermetic and reproducible over time.
We introduce a single "fetch-url" "using" flavor to define tasks that
fetch single URLs and then re-expose that URL as an artifact. Powering
this is a new, minimal "fetch" Docker image that contains a
"fetch-content" Python script that does the work for us.
We have added tasks to fetch source archives used to build the GCC
toolchains.
Fetching remote content and re-exposing it as an artifact is not
very useful by itself: the value is in having tasks use those
artifacts.
We introduce a taskgraph transform that allows tasks to define an
array of "fetches." Each entry corresponds to the name of a "fetch"
task kind. When present, the corresponding "fetch" task is added as a
dependency. And the task ID and artifact path from that "fetch" task
is added to the MOZ_FETCHES environment variable of the task depending
on it. Our "fetch-content" script has a "task-artifacts"
sub-command that tasks can execute to perform retrieval of all
artifacts listed in MOZ_FETCHES.
To prove all of this works, the code for fetching dependencies when
building GCC toolchains has been updated to use `fetch-content`. The
now-unused legacy code has been deleted.
This commit improves the reliability and efficiency of GCC toolchain
tasks. Dependencies now all come from task artifacts and should always
be available in the common case. In addition, `fetch-content` downloads
and extracts files concurrently. This makes it faster than the serial
application which we were previously using.
There are some things I don't like about this commit.
First, a new Docker image and Python script for downloading URLs feels
a bit heavyweight. The Docker image is definitely overkill as things
stand. I can eventually justify it because I want to implement support
for fetching and repackaging VCS repositories and for caching Debian
packages. These will require more packages than what I'm comfortable
installing on the base Debian image, therefore justifying a dedicated
image.
The `fetch-content static-url` sub-command could definitely be
implemented as a shell script. But Python is readily available and
is more pleasant to maintain than shell, so I wrote it in Python.
`fetch-content task-artifacts` is more advanced and writing it in
Python is more justified, IMO. FWIW, the script is Python 3 only,
which conveniently gives us access to `concurrent.futures`, which
facilitates concurrent download.
`fetch-content` also duplicates functionality found elsewhere.
generic-worker's task payload supports a "mounts" feature which
facilitates downloading remote content, including from a task
artifact. However, this feature doesn't exist on docker-worker.
So we have to implement downloading inside the task rather than
at the worker level. I concede that if all workers had generic-worker's
"mounts" feature and supported concurrent download, `fetch-content`
wouldn't need to exist.
`fetch-content` also duplicates functionality of
`mach artifact toolchain`. I probably could have used
`mach artifact toolchain` instead of writing
`fetch-content task-artifacts`. However, I didn't want to introduce
the requirement of a VCS checkout. `mach artifact toolchain` has its
origins in providing a feature to the build system. And "fetching
artifacts from tasks" is a more generic feature than that. I think
it should be implemented as a generic feature and not something that is
"toolchain" specific.
I think the best place for a generic "fetch content" feature is in
the worker, where content can be defined in the task payload. But as
explained above, that feature isn't universally available. The next
best place is probably run-task. run-task already performs generic,
very-early task preparation steps, such as performing a VCS checkout.
I would like to fold `fetch-content` into run-task and make it all
driven by environment variables. But run-task is currently Python 2
and achieving concurrency would involve a bit of programming (or
adding package dependencies). I may very well port run-task to Python
3 and then fold fetch-content into it. Or maybe we leave
`fetch-content` as a standalone script.
MozReview-Commit-ID: AGuTcwNcNJR
--HG--
extra : rebase_source : 4918b8c3bac53d63665006802054038bfbca0314
After this change, we consistently import GPG keys from files in
the GCC build scripts.
MozReview-Commit-ID: BcyvCQoGbMS
--HG--
extra : rebase_source : 657ccce8e242cabdfaff396fd0d6439754a3f364
python-zstandard's 0.9.1 source distribution contains a debian/
directory.
On Squeeze, producing a Debian package is straightforward.
On Wheezy, we need to hack up Build-Depends because Wheezy doesn't
have a package for the Hypothesis fuzzing library. This package is
only used for testing and our package building disables testing,
so we don't even need to further hack up the packaging to disable
tests.
MozReview-Commit-ID: 6raXjdzggCH
--HG--
extra : rebase_source : 672492a40d65df8430eb17ba033bcb1c0890b7df
This build target doesn't have LTO enabled on it (yet)
MozReview-Commit-ID: 56tAHMyvH7o
--HG--
extra : rebase_source : 90039cd8e97332e2ef8aad7908b8a04b2869f4a5
The crash reporter symbol files are the easiest cross-platform way to
find static initializers. While some types of static initializers (e.g.
__attribute__(constructor) functions) don't appear there in a notable
way, the static initializers we do care the most about for tracking do
(static initializers from C++ globals). As a matter of fact, there is
only a difference of 2 compared to the currently reported count of 125
on a linux64 build, so this is a good enough approximation. And allows
us to easily track the count on Android, OSX and Windows builds, which
we currently don't do.
The tricky part is that the symbol files are in
dist/crashreporter-symbols/$lib/$fileid/$lib.sym, and $fileid is hard to
figure out. There is a `fileid` tool in testing/tools, but it is a
target binary, meaning it's not available on cross builds (OSX,
Android).
So the simplest is just to gather the data while creating the symbol
files, which unfortunately requires to go through some hoops to make it
happen for just the files we care about.
--HG--
extra : rebase_source : 458fed1ffd6f9294eefef61f10ff7a284af0d986
for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1403
--HG--
extra : rebase_source : b7b7f34204bd06ba2baf92ae6d103f8c207214fb
for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1403
--HG--
extra : rebase_source : fc669df941b70b720a6eb4118ad36d6679a28d48
This relies on the fact that providing multiple --version-script
combines them all, so we effectively create a new symbol version
that has no global symbol, but hides the std:🧵:_M_start_thread
symbols.
This version script trick happens to work with BFD ld, gold, and lld.
The downside is that when providing multiple --version-script's, ld
doesn't want any of them to have no version at all. So for the libraries
that do already have a version script (through SYMBOLS_FILE), we use a
version where there used to be none, using the library name as the
version. Practically speaking, this binds the libraries a little closer
than they used to be, kind of non-flat namespace on OSX (which is the
default there), meaning the dynamic linker will actively want to use
symbols from those libraries instead of a system library that might
happen to have the same symbol name.
--HG--
extra : rebase_source : a7f672c35609d993849385ddb874ba791b34f929
Version 2.25.1's libiberty can choke on some symbols. That was fixed in
2.27. As of writing, the last version is 2.30. Conservatively go with
2.28.1, which is the same major version (2.28) as what is currently in
Debian stable.
--HG--
extra : rebase_source : 9e5ba94421a1568f662cfd98af7168ea1c890934
This relies on the fact that providing multiple --version-script
combines them all, so we effectively create a new symbol version
that has no global symbol, but hides as much std::* stuff as possible.
The added symbol script could use `extern "C++"` syntax and demangled
symbols but there is no guarantee the demangled symbols won't change.
Plus, it's not possible to match demangled symbols that have a return
type: they contain a space, and the only way to match that is to use
double quotes, which doesn't allow globs at the same time.
This version script trick happens to work with BFD ld, gold, and lld.
The downside is that when providing multiple --version-script's, ld
doesn't want any of them to have no version at all. So for the libraries
that do already have a version script (through SYMBOLS_FILE), we use a
version where there used to be none, using the library name as the
version. Practically speaking, this binds the libraries a little closer
than they used to be, kind of non-flat namespace on OSX (which is the
default there), meaning the dynamic linker will actively want to use
symbols from those libraries instead of a system library that might
happen to have the same symbol name.
--HG--
extra : rebase_source : 78adb64b90e75ebad203b8a647b305c9d7198d16
In the span of one week, both gmplib.org and multiprecision.org,
respective home of gmp and mpc have gone down. The latter is still down.
It turns out that all versions of gmp and mpfr we need are mirrored on
ftp.gnu.org, so we can just use that instead. For mpc, versions > 1.0
are on ftp.gnu.org, but not earlier versions.
The one mpc version <= 1.0 we do need is 0.8.2, and a copy of the exact
same archive, as per its sha256, which we're already checking per the
gcc build scripts, can be found on snapshot.debian.org. We lose gpg
validation on the way, but since we're already checking the sha256,
that's a fine tradeoff.
At least this unblocks changes to toolchains until multiprecision.org
comes back online.
--HG--
extra : rebase_source : f2bc67f8757655d99bfd9735b80d7205f7bfe47b
Nathan Froyd
Ted Mielczarek
Sebastian Hengst
Hiroyuki Ikezoe
Chris Peterson
Chris Manchester
svoisen
Mike Hommey
Andreea Pavel
Boris Chiou
Jan Beich
David Major
Margareta Eliza Balazs
Masatoshi Kimura
Csoregi Natalia
Emilio Cobos Álvarez
Bob Clary
Dan Mosedale
Daniel Varga
Gabriele Svelto
arthur.iakab
Cosmin Sabou
Tiberius Oros
Dorel Luca
Bogdan Tara
Narcis Beleuzu
Julian Seward
Tom Ritter
Dan Mosedale
Noemi Erli
David Heiberg
Jesse Schwartzentruber
Brindusan Cristian
Geoff Brown
Panos Astithas
Tudor-Gabriel Vîjială
Nika Layzell
Christian Holler
Mike Shal
Devika Sugathan
Mike Shal
Robert Bartlensky
Kartikaya Gupta
Dan Minor
Jacek Caban
Ivan Pozdeev
Marco Castelluccio
Gurzau Raul
Ciure Andrei
Ms2ger
Gregory Szorc
Simon Fraser
Dave Hunt
Henri Sivonen
Andrew Halberstadt
Xidorn Quan
Mike Hommey
Kris Maglione
Johan Lorenzo
Miko Mynttinen
shindli
Jonathan Watt
Chris AtLee
Sylvestre Ledru
Justin Wood
Sofia Carrillo
Andi-Bogdan Postelnicu