The main reason we look at the complete symbols table is that before bug
1541792, we needed to look at that table for _NSModule symbols.
In bug 1516228, we also made everything llvm-objdump to limit the
differences cross-platform, but that's not necessary anymore per the
previous change.
llvm-objdump doesn't support getting only the dynamic symbols table, so
we go back to what we were using before bug 1516228, namely readelf,
while preserving a code path to use the complete symbols table for the
networking test on libgkrust.a, which doesn't have a dynamic symbols
table.
With this change, check_binary goes from 45s to 0.2s on my machine.
Differential Revision: https://phabricator.services.mozilla.com/D84305
The rules we create in the backend for generated files use a stub file
for the real file generation, and we do need dependencies on that
specific target for the changes in bug 1264836 to work without races.
Differential Revision: https://phabricator.services.mozilla.com/D84311
Otherwise, crates that are shared between host and target are always
rebuilt because of the difference in rust flags.
Differential Revision: https://phabricator.services.mozilla.com/D83412
Doing this means that make will correctly understand that the underlying
program has changed and therefore it needs to trigger install rules.
Differential Revision: https://phabricator.services.mozilla.com/D83326
This makes dir/target, dir/target-objects, etc. depend on
dir/pre-compile, which ensures all items in the pre-compile pseudo-tier
are built before entering in the directory they are defined in, during
the compile tier. This removes the need for EXTRA_DEPS hacks.
Differential Revision: https://phabricator.services.mozilla.com/D83036
Generated files marked as "required during compile" are limited to a
number of extensions, most of which are source file types, so they get
generated when compiling the corresponding object file. Other types,
though, are currently handled via EXTRA_DEPS, which is clunky.
As of now, these other types are limited to link-related items, but more
would be useful (e.g. .inc files, which are only included in one
location).
This however works against the static analysis tasks (coverity and
clang-tidy), which currently rely on everything they need being created
via the export tier. That excludes using EXTRA_DEPS-based hacks.
We create a "pre-compile" tier, that is not used during a normal build,
but can be invoked manually, which the static analysis tasks will do.
Differential Revision: https://phabricator.services.mozilla.com/D83035
The previous logic would check the value of the `PYTHON3` environment variable, which is not set by the Firefox build system, so this build script tends to just use the `python3` at the tip of the `PATH`, regardless of which value you've configured for it. Instead, it should prefer to read from the `OBJDIR` if one is present.
Differential Revision: https://phabricator.services.mozilla.com/D82981
This makes it much simpler for the for the static JS services cache to store
and lookup interface IDs for components.
Differential Revision: https://phabricator.services.mozilla.com/D81416
... instead of a C file. This avoids issues with sccache (via dist or
caching), because sccache doesn't do anything with assembly files,
but with a C file, it doesn't know about the incbin-included data file.
This has the unfortunate side effect of requiring some manual work to
mark the file for safeseh on 32-bits Windows.
Differential Revision: https://phabricator.services.mozilla.com/D82814
This creates a general pytest plugin under "config/mozunit" which implements
the ability to mark tests as "slow". Slow tests will be skipped by default
unless "--run-slow" is passed in.
Differential Revision: https://phabricator.services.mozilla.com/D81402
This makes it much simpler for the for the static JS services cache to store
and lookup interface IDs for components.
Differential Revision: https://phabricator.services.mozilla.com/D81416
This is strictly a quality of life improvement when cross-compiling to
Windows targets. A common scenario is mounting an object directory
into a Windows VM, which requires additional VM configuration to
handle (absolute) symlinks. With this patch, `export NSDISTMODE=copy`
in a mozconfig sidesteps any such symlink issues.
Differential Revision: https://phabricator.services.mozilla.com/D77118
For a while, now, ICU data has been stored as raw read-only data in rather
than a separate loadable library. This means the parts of ICU that can load
such a module are not necessary.
Differential Revision: https://phabricator.services.mozilla.com/D82262
All the supported compilers support a GNU AS-like syntax, with only a
few details varying. It means we can use a single, simpler, way to
include the ICU data file, instead of 3 different ways, including one
that uses armasm64.exe, possibly wrapped with Wine.
Differential Revision: https://phabricator.services.mozilla.com/D82144
Because jar_maker is not in the libs tier, we also rename the libs-%
targets for l10n repacks to l10n-%, which make it clearer what they are
for.
And because multilocale.txt is both a GeneratedFile and a file that is
generated (and installed) via manual build rules, keeping it in the misc
target actually breaks building in toolkit/locales during l10n repacks,
so move it to libs for now.
Differential Revision: https://phabricator.services.mozilla.com/D81766
While ideally we'd just move all of them at the top-level, there are
other things that depend on them that wouldn't then get the right
dependencies. One of them is install steps in dist/something, but there
are others, and that's a rather long pole of things requiring many
changes along with chances to break things.
So instead, we go with a simpler and more limited approach, where we
still recurse into directories to run their export tier (to run whatever
else than GeneratedFiles they have), but ensure the GeneratedFiles we
moved at the top-level are built before recursing by making
directory/export depend on them.
Differential Revision: https://phabricator.services.mozilla.com/D80608
While ideally we'd just move all of them at the top-level, there are
other things that depend on them that wouldn't then get the right
dependencies. One of them is install steps in dist/something, but there
are others, and that's a rather long pole of things requiring many
changes along with chances to break things.
So instead, we go with a simpler and more limited approach, where we
still recurse into directories to run their export tier (to run whatever
else than GeneratedFiles they have), but ensure the GeneratedFiles we
moved at the top-level are built before recursing by making
directory/export depend on them.
Differential Revision: https://phabricator.services.mozilla.com/D80608
While ideally we'd just move all of them at the top-level, there are
other things that depend on them that wouldn't then get the right
dependencies. One of them is install steps in dist/something, but there
are others, and that's a rather long pole of things requiring many
changes along with chances to break things.
So instead, we go with a simpler and more limited approach, where we
still recurse into directories to run their export tier (to run whatever
else than GeneratedFiles they have), but ensure the GeneratedFiles we
moved at the top-level are built before recursing by making
directory/export depend on them.
Differential Revision: https://phabricator.services.mozilla.com/D80608
They were necessary some time ago, but:
- there is no js/xpconnect/src/export target anymore,
- accessibility/xpcom/export had a dependency on xpcom/xpidl/export for
the ply cache files, but they're not used as of bug 1633156.
- the recursive make backend was adding automatic dependencies on
xpcom/xpidl/export for the ply cache files.
Differential Revision: https://phabricator.services.mozilla.com/D80150
Replace the duplicate lists in mozglue/baseprofiler/public and js/public with
a shared list. Add this list to both moz.build files so it is published twice
which simplifies supporting different standalone configurations.
Differential Revision: https://phabricator.services.mozilla.com/D79520
IMPACTED_TESTS was added in bug 1184405, but is under-utilized in current moz.build
annotations; also bug 1516847 removed use of IMPACTED_TESTS in 'mach test'. This
patch removes remaining support for IMPACTED_TESTS.
Differential Revision: https://phabricator.services.mozilla.com/D79141
Extremely common file paths on macOS like `/Users/...` are interpreted
as `/U...` flags by `clang-cl`. This is so common that there's a
`-Wslash-u-filename` warning. Ensure that file paths are considered
paths when compiling by terminating options with `--`.
This commit handles everything except assembler invocations, because
at least `nasm` doesn't handle `--`.
Differential Revision: https://phabricator.services.mozilla.com/D77119
This is strictly a quality of life improvement when cross-compiling to
Windows targets. A common scenario is mounting an object directory
into a Windows VM, which requires additional VM configuration to
handle (absolute) symlinks. With this patch, `export NSDISTMODE=copy`
in a mozconfig sidesteps any such symlink issues.
Differential Revision: https://phabricator.services.mozilla.com/D77118
Also update documentation to suggest using the `GeneratedFile` template rather than directly referencing `GENERATED_FILES` where possible.
Differential Revision: https://phabricator.services.mozilla.com/D77496
l10n-merge creates a full merge dir for a while now, let's
simplify the build logic to only read from that directory
during repacks and langpacks.
Differential Revision: https://phabricator.services.mozilla.com/D77023
I haven't been able to confirm that this would actually fix
`--with-system-icu`, but since the new regexp
engine (`js/src/new-regexp`) explicitly includes this file and this file
isn't already present in system-headers...it seems very likely that this
change will fix `--with-system-icu`.
Differential Revision: https://phabricator.services.mozilla.com/D75809
iOS support for Gecko has not been tested in years and is most probably
out of date. The build system part of it, specifically the checks in
build/autoconf/ios.m4, are not trivial to port to python configure, and
they prevent other things from moving to python configure (because some
of them change value when MOZ_IOS is set).
The code is left alone, although it could probably be stripped off as
well, but I'll leave that as an exercise for someone else.
Differential Revision: https://phabricator.services.mozilla.com/D75463
`ply`, [by design](https://github.com/dabeaz/ply/issues/79), does not produce reproducible table files; hence bug 1633156. (Note that this was *always* true, but only became a problem once we switched to Python 3, which has more unpredictable dict iteration order than Python 2.7, at least prior to [3.7](https://docs.python.org/3/whatsnew/3.7.html#summary-release-highlights).)
In any other circumstance I would consider submitting a patch to `ply` to fix this, but as of the [in-progress version 4.0 of the library](https://github.com/dabeaz/ply/blob/master/CHANGES), it doesn't even emit this cached data any more, and indeed the [latest version of the code](1fac9fed64/ply) doesn't even call `open()` at all except to do logging or to read the text data to be parsed from `stdin`. So if we were going to pin our future on `ply` and upgrade to later versions of the library in the future, we would have to live in a world where `ply` doesn't generate cached table files for us anyway.
Emitting the cached table files so later build steps can consume them is an "optimization", but it's not clear exactly how much actual value that optimization provides overall. Quoth the `CHANGES` file from that repository:
```
PLY no longer writes cached table files. Honestly, the use of
the cached files made more sense when I was developing PLY on
my 200Mhz PC in 2001. It's not as much as an issue now. For small
to medium sized grammars, PLY should be almost instantaneous.
```
In practice, I have found this to be true; namely, `./mach build pre-export export` takes just about as long on my machine after this patch as it did before, and in a try push I performed, there's no noticeable performance regression from applying this patch. In local testing I also found that generating the LALR tables in calls to `yacc()` takes about 0.01s on my machine generally, and we generate these tables a couple dozen times total over the course of the `export` tier now. This isn't *nothing*, but in my opinion it's also not nearly long enough where it would be a concern given how long `export` already takes.
That `CHANGES` file also stresses that if caching this data is important, we have the option of doing so via `pickle`. If and when we decide that re-enabling this optimization is valuable for us, we should take control of this process and perform the generation in such a way that we can guarantee reproducibility.
Differential Revision: https://phabricator.services.mozilla.com/D73484
Linking Rust code against the updatecommon library is not well supported by our build system, but it should be possible. The changes to updatecommon's moz.build will cause updatecommon.lib to be created, which Rust can link against. The build.rs script tells cargo which shared Windows libraries are needed for linking, and where updatecommon.lib can be found. The change to config/recurse.mk enforces the dependency of the update agent on updatecommon, so that we do not attempt to link the agent before updatecommon has been built.
Differential Revision: https://phabricator.services.mozilla.com/D70207
For native sanitizer builds, we currently do not pass the linker flags
to cargo, as they were causing crashes in some build scripts. Without
this, however, the linker is unable to find libstdc++. Instead, do
tell cargo to use the linker wrapper, but omit the problematic flags
from MOZ_CARGO_WRAP_LDFLAGS.
Differential Revision: https://phabricator.services.mozilla.com/D70354
For now this generates just CACHE_IR_OPS and CACHE_IR_SHARED_OPS in CacheIROpsGenerated.h
but the plan is to use this to generate parts of the IR writer and compiler/transpiler
interface. The spewer could also potentially be improved now that each operand has a name
and more precise type.
Generating the IR writer will likely happen incrementally so that will give us
another chance to double check the precise types match what's in the YAML file.
Differential Revision: https://phabricator.services.mozilla.com/D70995
--HG--
extra : moz-landing-system : lando
Most other generated SpiderMonkey files have the 'Generated' suffix so let's
follow that convention.
Differential Revision: https://phabricator.services.mozilla.com/D70993
--HG--
extra : moz-landing-system : lando
They all fail for the same reason: the sanitizer runtime in compiler-rt installs an interceptor for `getsockname` which then contains a call to the real implementation.
Differential Revision: https://phabricator.services.mozilla.com/D70889
--HG--
extra : moz-landing-system : lando
Also enable all config/tests with python3. unit-nsinstall.py was the
last one that didn't pass with python 3.
Switch the test to using @unittest.skipIf and disable the subprocess
test because we purposely broke running nsinstall.py independently with
python 2.
Differential Revision: https://phabricator.services.mozilla.com/D70160
--HG--
extra : moz-landing-system : lando
Since the test goes through all moz.build files disregarding DIRS and
the conditions that may disable directories, in some cases, moz.builds
can fail to be evaluated properly because of missing variables in
config.status. This time (because it's not the first), it's
LLVM_DLLTOOL.
After fixing that, it turns out many of the files/directories pointed to
by Files() directives were removed or moved.
While here, make the test script python3-ready.
Differential Revision: https://phabricator.services.mozilla.com/D70157
--HG--
extra : moz-landing-system : lando
This needs a few adjustments to the autobinscope script because running
binscope currently creates an HTML file in the binscope directory, and
when multiple binscopes run at the same time (which happens during the
build with the changes to run it on all executables and libraries), all
but one fail to open the HTML file for write access.
So add a flag to create that file in a temporary directory.
While here, remove log_file_path, which hasn't been used since
bug 1448306.
Differential Revision: https://phabricator.services.mozilla.com/D67422
--HG--
extra : moz-landing-system : lando
Because most of the recent changes to irregexp were patches I contributed myself, we barely need to change any of the shim code.
The only notable change is the addition of '#define COMPILING_IRREGEXP_FOR_EXTERNAL_EMBEDDER'. This is the solution that Jakob Gruber and I eventually came up with for the question of what to do with awkward V8 code that SM doesn't want. For example, NativeRegExpMacroAssembler::Match (in regexp-macro-assembler.cc) gets down in the muck with the internal details of V8's String implementation. It would be most convenient for SM if that function just didn't exist; we aren't going to use it, and we don't want to have to define a bunch of unused string API gunk in our shim. The answer is to wrap functions we don't need in "#ifndef COMPILING_IRREGEXP_FOR_EXTERNAL_EMBEDDER", which solves our problem and is minimally disruptive upstream.
Depends on D67718
Differential Revision: https://phabricator.services.mozilla.com/D67719
--HG--
extra : moz-landing-system : lando
This needs a few adjustments to the autobinscope script because running
binscope currently creates an HTML file in the binscope directory, and
when multiple binscopes run at the same time (which happens during the
build with the changes to run it on all executables and libraries), all
but one fail to open the HTML file for write access.
So add a flag to create that file in a temporary directory.
While here, remove log_file_path, which hasn't been used since
bug 1448306.
Differential Revision: https://phabricator.services.mozilla.com/D67422
--HG--
extra : moz-landing-system : lando
Dictionary iteration under Python 3 is in an inherently unpredictable order, and while we try to keep DEFINES ordered through the use of OrderedDicts, if at any point we populate DEFINES directly or indirectly while iterating through the contents of a non-ordered dictionary, the order of the DEFINES (and therefore the contents of the output Makefile) will be nondeterministic as well. This patch makes a number of changes to ensure that we only ever populate DEFINES in a deterministic fashion. (Note that in Python 3.7 and later, the built-in dict class actually has deterministic ordering, so these changes are technically only necessary until our minimum Python version becomes 3.7.)
Differential Revision: https://phabricator.services.mozilla.com/D66089
--HG--
extra : moz-landing-system : lando
Mozilla Releng Treescript
Mike Hommey
Butkovits Atila
Narcis Beleuzu
Nathan Froyd
Tom Prince
Ricky Stewart
Kris Maglione
Csoregi Natalia
Andrew Halberstadt
Axel Hecht
Nick Alexander
André Bargull
Emilio Cobos Álvarez
Coroiu Cristina
Dorel Luca
Ted Campbell
Geoff Brown
Bogdan Tara
Kirk Steuber
Jamie Nicol
Nika Layzell
Jan de Mooij
David Major
Jan Beich
Sylvestre Ledru
Michal Novotny
Iain Ireland
Oana Pop Rus
William Lachance
Michael Woerister
Anthony Ramine
Mike Shal