Historically, client.mk was not invoked with -jn because it would create
race conditions, but that was actually mostly solved by the addition of
`.NOTPARALLEL` in bug 422986, although the mechanism of adding -jn via
`MOZ_MAKE_FLAGS` or `MOZ_PARALLEL_BUILD` has continued well past that.
Nowadays, client.mk is only invoked by mach (it will even bail out if
that's not the case) and only has one target (`build`) and no
dependencies.
This means we don't need to rely on `MOZ_PARALLEL_BUILD` to pass `-jn` in
some cases, and can just always invoke `make -f client.mk` with `-jn`, even
when we just want no parallelism, in which case we can use `-j1`.
This, in turn, allows to remove the extra allow_parallel argument to
`_run_make`, and only rely on `num_jobs`, and to remove some of the
multiple ways the `n` in `-jn` could be set.
Differential Revision: https://phabricator.services.mozilla.com/D124729
Automatically generated path that adds flag `REQUIRES_UNIFIED_BUILD = True` to `moz.build`
when the module governed by the build config file is not buildable outside on the unified environment.
This needs to be done in order to have a hybrid build system that adds the possibility of combing
unified build components with ones that are built outside of the unified eco system.
Differential Revision: https://phabricator.services.mozilla.com/D122345
MKCSHLIB, used to build WASM_LIBRARY, refers to DSO_LDOPTS, which refers
to IMPORT_LIBRARY on mingw builds, but for WASM_LIBRARY, IMPORT_LIBRARY
is never set by the build backend.
Differential Revision: https://phabricator.services.mozilla.com/D123540
The build system assumes the target compiler is of the same type as the
wasm compiler, but that's not true for wasm32-wasi, which we compile
with clang, while targeting windows, which we compile with clang-cl.
We handle the duality for host/target in a compiler-specific way, but
here, the wasm compiler is always going to be clang.
Differential Revision: https://phabricator.services.mozilla.com/D123041
The build system assumes the target compiler is of the same type as the
wasm compiler, but that's not true for wasm32-wasi, which we compile
with clang, while targeting windows, which we compile with clang-cl.
We handle the duality for host/target in a compiler-specific way, but
here, the wasm compiler is always going to be clang.
Differential Revision: https://phabricator.services.mozilla.com/D123041
The build system assumes the target compiler is of the same type as the
wasm compiler, but that's not true for wasm32-wasi, which we compile
with clang, while targeting windows, which we compile with clang-cl.
We handle the duality for host/target in a compiler-specific way, but
here, the wasm compiler is always going to be clang.
Differential Revision: https://phabricator.services.mozilla.com/D123041
Except for the following two files, no other header file is still including
"jsapi.h". Add a rule to "check_spidermonkey_style.py" to make sure we don't
regress this progress.
Exceptions:
- "vm/Compartment-inl.h" still needs to include "jsapi.h" for `JS::InformalValueTypeName()`.
- "jsapi-tests/tests.h" still needs to include "jsapi.h" for `JS_ValueToSource()`.
Differential Revision: https://phabricator.services.mozilla.com/D123209
They are equivalent, except for the fact that MACOSX_DEPLOYMENT_TARGET
may apply to more processes in a normal build. In practice, all the
processes that matter are covered through compiler flags.
On the opposite end, MACOSX_DEPLOYMENT_TARGET isn't necessary passed in
all cases (like clangd, mach static-analysis, etc.), while compiler
flags are.
Differential Revision: https://phabricator.services.mozilla.com/D122145
Sanitizers try to intercept __tls_get_addr to find DTLS blocks. In some
cases, they fail miserably with a crash. And in the cases they don't
crash, they don't seem to actually work properly anyways.
This affects both build time when running host programs compiled with
sanitizer flags (this only actually affects rust build
scripts/procedural macros on non-cross builds, not C/C++ host programs),
and execution time, e.g. when running tests.
Differential Revision: https://phabricator.services.mozilla.com/D120574
The `-/*` filter also removes the alias (`%%ALIAS`) and parent locale (`%%Parent`)
links, so we have to manually add them back when we're using `-/*`.
The ICU docs warn about this issue, but unfortunately that warning wasn't
present when we originally created our filter file, so this bug went unnoticed
for a long time.
Differential Revision: https://phabricator.services.mozilla.com/D116981
It's not practically make a difference (the flags we end up setting it
to don't do anything when linking all the wasm files into one).
Differential Revision: https://phabricator.services.mozilla.com/D116437
This last part to support "calendar" requires a bit more code.
`GetCalendarDisplayName()` performs the following steps:
1. Validate the input matches the `type` nonterminal, per
CanonicalCodeForDisplayNames, step 4.a.
2. Then convert it to lower case, because ICU expects lower case names and it
needs to be in lower case for `LanguageTag::replaceUnicodeExtensionType()`.
3. Replace calendar name aliases with the preferred name, e.g. use "ethioaa"
instead of "ethiopic-amete-alem".
4. Check the input isn't too large, just in case ICU somehow doesn't properly
handle large string inputs.
5. Retrieve the old-style value of the calendar name, for example "gregorian"
for the calendar name "gregory".
6. And finally call `uldn_keyValueDisplayName` to search for the localised name.
Our ICU data file doesn't yet include calendar names. To include calendar names
we need to enable them in "data_filter.json", rebuild the ICU data file, and
then reapply the update for tzdata 2021a. (Reapplying the tzdata changed
"intl/tzdata/GIT-INFO", but otherwise didn't change any time zone files.)
Adding the calendar names to the ICU data file results in a slight increase of
the overall data file size (79,184 bytes).
The existing "calendar.js" test file was renamed to "month-calendar.js", so
the test file for the new "calendar" type can be named "calendar.js".
Differential Revision: https://phabricator.services.mozilla.com/D116008
This adds a fallible factory method to create new NumberFormat instances. This
allows us to report initialization errors at time of initialization, rather than
when format is called, and remove internal checks in the implementation for
successful initialization.
The existing fluent code assumes that creating a NumberFormat instance always
succeeds. This patch updates that code to handle failures.
Differential Revision: https://phabricator.services.mozilla.com/D114593
This temporarily adds intl::NumberFormatterSkeleton to the public API so that
we can use it in PluralRules. This allows us to remove the SpiderMonkey
NumberFormatterSkeleton implementation when we convert SpiderMonkey to use
intl::NumberFormat.
We can remove intl::NumberFormatterSkeleton from the public API when we
implement intl::PluralRules in Bug 1704509.
Differential Revision: https://phabricator.services.mozilla.com/D111887
This adds a fallible factory method to create new NumberFormat instances. This
allows us to report initialization errors at time of initialization, rather than
when format is called, and remove internal checks in the implementation for
successful initialization.
The existing fluent code assumes that creating a NumberFormat instance always
succeeds. This patch updates that code to handle failures.
Differential Revision: https://phabricator.services.mozilla.com/D114593
This temporarily adds intl::NumberFormatterSkeleton to the public API so that
we can use it in PluralRules. This allows us to remove the SpiderMonkey
NumberFormatterSkeleton implementation when we convert SpiderMonkey to use
intl::NumberFormat.
We can remove intl::NumberFormatterSkeleton from the public API when we
implement intl::PluralRules in Bug 1704509.
Differential Revision: https://phabricator.services.mozilla.com/D111887
- We can and actually should remove the -fsanitize=% flags consistently
on all cargo targets, not only programs, but because of limitations in
cargo we can't really do that everywhere. We do need it for TSAN,
though.
- Because of the above, we actually don't need to single-out NATIVE_TSAN
builds anymore.
- The comment regarding the initial filtering of MOZ_CARGO_WRAP_LDFLAGS
somehow got separated from where it's happening, so we move it back to
where it belongs.
Differential Revision: https://phabricator.services.mozilla.com/D114579
Credit goes to dmajor.
Rust gets LLVM 12 - https://github.com/rust-lang/rust/pull/81451.
The PGO format changed between 11 and 12, and builds will have poor optimization if we mix versions in xLTO.
The above PR has landed on Rust nightly 1.52.
Until we move to 1.52 nightly we should have this landed.
In the meantime, for unblocking try builds, we can disable value-PGO in Rust.
Differential Revision: https://phabricator.services.mozilla.com/D113327
When "build targets" are used, the sccache server isn't
being used/integrated properly with make.
By moving rusttests to a separate build "application",
`client.mk` is invoked normally, and sccache is happy.
* Move rusttests to a new build "application".
* Move and denormalize rusttests mozconfigs. The new rusttests
mozconfigs should expand to be mostly identical to their old
versions, except with `MOZ_AUTOMATION_*=0` and
`--enable-application=tools/rusttests`.
They will differ a bit from their old variants because some
of them originally didn't source from their
respective `$app_mozconfigs/common`.
* Modify `baseconfig.mk` to change tiers if running rusttests.
* Adjust CI config to use new "rusttests" build application.
Differential Revision: https://phabricator.services.mozilla.com/D106294
While we could change qcms, encoding_rs and packed_simd to not emit
RUSTC_BOOTSTRAP on newer versions of rust, like we do for gkrust-shared,
it's not worth the effort (especially for those that are vendored).
Differential Revision: https://phabricator.services.mozilla.com/D105423
Bug 1690930 added sysroots that can be bootstrapped. With this change,
we allow --enable-bootstrap=install to pull the right sysroot for the
configured target, and --enable-bootstrap to update it if it was already
there.
Differential Revision: https://phabricator.services.mozilla.com/D104797
Back when the flag was added, the libffi source code didn't build for
ARM with the clang integrated assembler. That has been fixed with the
upgrade to libffi v3.3 in bug 1659906, so we can use the integrated
assembler now.
Differential Revision: https://phabricator.services.mozilla.com/D102261
* We add a config option for setting the number of content services;
* We add a config option to indicate whether content services should be isolated.
This one is just a `project_flag` since it doesn't really need the ability to
be overridden; it's something whose default we would want to flip when the
time comes;
* We set a dependency so that mobile/android/base/pre-export is executed;
* We add the `gen_from_jinja.py` script which is mostly just a dumb shim that
takes the input template and the config arguments, instantiates jinja,
generates the final output, and dumps it to the output fd;
* We add the requisite `moz.build` statements to generate the manifest overlay
and the service definitions;
* We update `build.gradle` so that Gradle knows to look for the generated files
when building the apk.
Differential Revision: https://phabricator.services.mozilla.com/D82577
* We add a config option for setting the number of content services;
* We add a config option to indicate whether content services should be isolated.
This one is just a `project_flag` since it doesn't really need the ability to
be overridden; it's something whose default we would want to flip when the
time comes;
* We set a dependency so that mobile/android/base/pre-export is executed;
* We add the `gen_from_jinja.py` script which is mostly just a dumb shim that
takes the input template and the config arguments, instantiates jinja,
generates the final output, and dumps it to the output fd;
* We add the requisite `moz.build` statements to generate the manifest overlay
and the service definitions;
* We update `build.gradle` so that Gradle knows to look for the generated files
when building the apk.
Differential Revision: https://phabricator.services.mozilla.com/D82577
With clang 12, win32 debug tests run out of stack in the 1000-deep layout crashtests.
Usually when this type of thing happens, it's because the compiler made a drastically heavier inlining decision along the recursive part of the stack, with a difference in the hundreds of bytes, and we pay that price of that space 1000 times. In those cases we normally try to snip out the heavy piece with a MOZ_NEVER_INLINE.
However, in this case, the difference between old and new compilers was only something like 16 bytes of the cyclical piece. This suggests that we're already up pretty close against the limit, and we're only a bad day away from failing tests even with existing clang. This only happens in debug builds since they have lots of logging helpers on the stack. Release builds are still well under the limit.
So I propose to increase the stack limit in debug. While we normally have to be pretty careful about address space on win32, it should be less of a concern with these non-shipping builds.
Differential Revision: https://phabricator.services.mozilla.com/D101196
Also rename `branchIfNegativeBigInt` to `branchIfBigIntIsNegative` to match
the other branch methods from part 2 and part 8.
Differential Revision: https://phabricator.services.mozilla.com/D98158
Also rename `branchIfNegativeBigInt` to `branchIfBigIntIsNegative` to match
the other branch methods from part 2 and part 8.
Differential Revision: https://phabricator.services.mozilla.com/D98158
On a very parallel debug build, I see a long time just waiting for
bindgen / style compilation / geckoservo.
Turns out that a bunch of this is just proc macros / build scripts.
Optimizing it saves between 10 and 17 seconds of my debug build. We
might want to consider running bindgen much like cbindgen rather than
rebuilding it all the time, which should help a lot more, but my guess
is that this should still help with the pretty hot custom derives that
the style crate runs.
This needs rust 1.41, so the requirement for tools/crashreporter needs
to be bumped as a consequence. To make things simpler, it was bumped
to 1.47 while we're at it.
Differential Revision: https://phabricator.services.mozilla.com/D98366
* Bumps the tsan toolchain to rust-nightly-2020-11-14 that has my patches to make -Zbuild-std work in vendored environments:
* https://github.com/rust-lang/cargo/pull/8834
* https://github.com/rust-lang/rust/pull/78790
* Passes -Zbuild-std to cargo when MOZ_TSAN is defined (mk_add_options --enable-thread-sanitizer)
* Removes generic Rust supressions and adds much more specific ones
* One presumed upstream false positive from tsan not understanding the code
* One actual upstream bug tsan found (yay!)
* One new real issue uncovered
* One issue that probably already existed intermittently but I happened to hit
Differential Revision: https://phabricator.services.mozilla.com/D97165
This makes --enable-thread-sanitizer turn on Rust tsan (-Zsanitizer=thread).
This requires changing SpiderMonkey tsan to use the tsan rust nightly.
In future changes, more Rust tsan integration will key off of MOZ_TSAN.
Differential Revision: https://phabricator.services.mozilla.com/D96453
This makes --enable-thread-sanitizer turn on Rust tsan (-Zsanitizer=thread).
This requires changing SpiderMonkey tsan to use the tsan rust nightly.
In future changes, more Rust tsan integration will key off of MOZ_TSAN.
Differential Revision: https://phabricator.services.mozilla.com/D96453
These files were omitted from the original patch because reformatting them required some manual intervention in order to avoid breaking unit tests. Generally the `noqa` lines were already there and just needed to be moved from one line to another (due to the reformatting by `black`), but sometimes `black` saw fit to move a bunch of stuff all onto one line, requiring me to introduce new `noqa` lines.
Besides the autoformat by `black` and some manual fixups, this patch contains no other changes.
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94052
Depends on D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
These files were omitted from the original patch because reformatting them required some manual intervention in order to avoid breaking unit tests. Generally the `noqa` lines were already there and just needed to be moved from one line to another (due to the reformatting by `black`), but sometimes `black` saw fit to move a bunch of stuff all onto one line, requiring me to introduce new `noqa` lines.
Besides the autoformat by `black` and some manual fixups, this patch contains no other changes.
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94052
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
The support code for Android, which doesn't support shm_open and can't
use the memfd backend because of issues with its SELinux policy (see bug
1670277), has been reorganized to reflect that we'll always use its own
API, ashmem, in that case.
Differential Revision: https://phabricator.services.mozilla.com/D90605
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
The support code for Android, which doesn't support shm_open and can't
use the memfd backend because of issues with its SELinux policy (see bug
1670277), has been reorganized to reflect that we'll always use its own
API, ashmem, in that case.
Differential Revision: https://phabricator.services.mozilla.com/D90605
These files were omitted from the original patch because reformatting them required some manual intervention in order to avoid breaking unit tests. Generally the `noqa` lines were already there and just needed to be moved from one line to another (due to the reformatting by `black`), but sometimes `black` saw fit to move a bunch of stuff all onto one line, requiring me to introduce new `noqa` lines.
Besides the autoformat by `black` and some manual fixups, this patch contains no other changes.
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94052
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
This avoids a sort of duplication of work between both, because the
linker will eventually LTO-compile everything, so we technically don't
really need the extra step of the rust compiler doing an intermediate
LTO on the static libraries it produces.
Differential Revision: https://phabricator.services.mozilla.com/D94224
Mike Hommey
Mozilla Releng Treescript
Andreea Pavel
shravanrn@gmail.com
Dzmitry Malyshau
Yury Delendik
Andi-Bogdan Postelnicu
Cosmin Sabou
Csoregi Natalia
André Bargull
Caroline Cullen
Mike Hommey
Sandor Molnar
Henri Sivonen
Noemi Erli
Butkovits Atila
Steve Fink
carolinecullen
Chris Fallin
Agi Sferro
Dan Minor
Brindusan Cristian
Kagami Sascha Rosylight
Dorel Luca
Iulian Moraru
Nick Alexander
Deian Stefan
Jesse Schwartzentruber
Christian Holler
Alexandru Michis
Mitchell Hentges
Masatoshi Kimura
Aaron Klotz
David Major
Kartik Gautam
Emilio Cobos Álvarez
Makoto Kato
Martin Stransky
Alexis Beingessner
Philipp Ammann
Ricky Stewart
Tom Schuster
Bogdan Tara
Jed Davis