This avoids a sort of duplication of work between both, because the
linker will eventually LTO-compile everything, so we technically don't
really need the extra step of the rust compiler doing an intermediate
LTO on the static libraries it produces.
Differential Revision: https://phabricator.services.mozilla.com/D94224
Rustc >= 1.44 changed the file names of the static libraries it
produces with -windows-gnu targets, to match that of mingw clang/gcc.
Considering we still build on 1.43, the best fix would be to derive the
prefix/suffix based on the version of rust, but that actually turns into
a hard-to-solve problem because of configure tests for bindgen also
depending on the prefix/suffix value to be known.
On the other hand, we're soon due to an update to 1.47, so the simpler
solution is to just push mingw builds to require 1.44 (settling for the
smallest upgrade possible for now) and to remove the split between C and
rust library prefix/suffixes.
Differential Revision: https://phabricator.services.mozilla.com/D93726
The `clobber` targets are superseded by `mach clobber`, so we don't need them for any reason. The `clean` target is meant to get you to a post-`configure` state, but it doesn't really work, and if it's necessary for you to be in that state for some reason you can just clobber and re-`configure`, so it doesn't seem worth it to get it working again. Instead, delete all of them. Also delete `everything` which is not useful when `clobber` doesn't exist.
Differential Revision: https://phabricator.services.mozilla.com/D93514
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
Differential Revision: https://phabricator.services.mozilla.com/D90605
It is only really used in js/src/devtools/rootAnalysis/Makefile.in,
and even there, the way it is used seems wrong, so fix that at the
same time (binaries have been linked into $DIST/bin directly for a
while).
Differential Revision: https://phabricator.services.mozilla.com/D92721
Bug 1573566 moved libxul from toolkit/library to toolkit/library/build,
and that should be reflected in config/recurse.mk.
It's amazing the race condition hasn't caused problems earlier than now.
Differential Revision: https://phabricator.services.mozilla.com/D92708
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
Differential Revision: https://phabricator.services.mozilla.com/D90605
Bug 1645986 solved the problem for most generated files by moving their
rules to the top-level, but we're going to add rules that will end up in
subdirectories, so we have to solve the same problem again, in the
subdirectories.
Differential Revision: https://phabricator.services.mozilla.com/D88389
Somehow we have had it be a general define, but it has not mattered...at
least it won't matter until we upgrade to clang 11, at which point the
compiler will start to complain at us about the codepaths this define
activates.
Differential Revision: https://phabricator.services.mozilla.com/D88056
All the patches previously applied, except the one from bug 1279096,
are either irrelevant (as pertaining to changes to the upstream build
system we don't use anymore), were applied upstream, or the issue they
fixed were fixed differently upstream.
Two additional patches, sent upstream as
https://github.com/libffi/libffi/pull/579 and
https://github.com/libffi/libffi/pull/580, are needed to fix our build
with, respectively, mingw-clang and GCC.
Our build system is adjusted according to upstream's configure.ac
and configure.host.
Differential Revision: https://phabricator.services.mozilla.com/D87824
All the patches previously applied, except the one from bug 1279096,
are either irrelevant (as pertaining to changes to the upstream build
system we don't use anymore), were applied upstream, or the issue they
fixed were fixed differently upstream.
Two additional patches, sent upstream as
https://github.com/libffi/libffi/pull/579 and
https://github.com/libffi/libffi/pull/580, are needed to fix our build
with, respectively, mingw-clang and GCC.
Our build system is adjusted according to upstream's configure.ac
and configure.host.
Differential Revision: https://phabricator.services.mozilla.com/D87824
This resolves a long-standing issue in development where `mach artifact` (and therefore `mach bootstrap`) would fail unpredictably if you had dirty, but ignored, files in your checkout. Resolving this problem often required unwieldy `hg purge`/`git ignore` incantations that are easy to get wrong.
This patch addresses the problem by doing what we "should" have been doing all along, and consulting the VCS to list tracked files rather than listing EVERY file on disk and applying heuristics to determine whether they should be included in the hash.
Differential Revision: https://phabricator.services.mozilla.com/D86780
This makes us use one less tool from MSVC, and removes one more use of wine
in cross builds.
We replace the call to either rc/llvm-rc or windres with a wrapper script.
While the script is not strictly needed for the latter, we use a wrapper
in that case anyway because it's one step towards fixing bug 1498414.
For llvm-rc, however, we need a wrapper because llvm-rc doesn't preprocess
on its own, so the wrapper does that too.
The wrapper script also allows to deal with the default flags passed to
llvm-rc or windres, rather than inherit them from old-configure.
We also need to explicitly pass the codepage to llvm-rc, which was not
necessary with rc (presumably, llvm-rc has a different default).
While here, remove the unused WINDRES subst from js/src/old-configure.in.
Also, while here, we remove --use-temp-file, because as described in the
linked bug and in the windres manual page, it was used to work around bugs
on Windows 98 and earlier.
Differential Revision: https://phabricator.services.mozilla.com/D86312
We are currently relying on two different ways to insert side-by-side
manifests in binaries on Windows: through resource files, or through
the use of MT. The latter is not supported on mingw builds, which is
not great.
Link.exe has options to add a manifest at link time without relying on
either method above, but that's not supported on mingw either.
So the best we can do is to move everything to using resource files.
This also avoids using MT, which, on cross builds, requires using wine.
Ideally, the manifests would be declared in moz.build, but that
complicates things for cases like TestDllInterceptor, where there are
multiple binaries in the same directory, but only one of them needs the
manifest. This keeps the status quo of getting the manifest
automatically from the source directory.
Differential Revision: https://phabricator.services.mozilla.com/D85382
We're currently not using the feature, and host programs ought only
to be used during the build, so I don't expect the feature to ever
be necessary.
Differential Revision: https://phabricator.services.mozilla.com/D86167
The resource file is always generated so being able to configure its name
is not useful. On the other hand, the way things are currently implemented,
the lack of RESFILE also makes RCFILE ignored, which we fix at the same
time.
And remove a spurious RESFILE in widget/windows/moz.build, where no binary
is produced, which means RESFILE had no meaning.
Differential Revision: https://phabricator.services.mozilla.com/D86154
Because while the original perl script was added to add version info
to Windows binaries, it does more and will do even more with upcoming
changes.
Differential Revision: https://phabricator.services.mozilla.com/D86153
It was only ever set to the same value as its default, except in
comm-central, where it is unset, but in directories that now don't link
anything (they did back when binary components were a thing).
Differential Revision: https://phabricator.services.mozilla.com/D85381
We are currently relying on two different ways to insert side-by-side
manifests in binaries on Windows: through resource files, or through
the use of MT. The latter is not supported on mingw builds, which is
not great.
Link.exe has options to add a manifest at link time without relying on
either method above, but that's not supported on mingw either.
So the best we can do is to move everything to using resource files.
This also avoids using MT, which, on cross builds, requires using wine.
Ideally, the manifests would be declared in moz.build, but that
complicates things for cases like TestDllInterceptor, where there are
multiple binaries in the same directory, but only one of them needs the
manifest. This keeps the status quo of getting the manifest
automatically from the source directory.
Differential Revision: https://phabricator.services.mozilla.com/D85382
We're currently not using the feature, and host programs ought only
to be used during the build, so I don't expect the feature to ever
be necessary.
Differential Revision: https://phabricator.services.mozilla.com/D86167
The resource file is always generated so being able to configure its name
is not useful. On the other hand, the way things are currently implemented,
the lack of RESFILE also makes RCFILE ignored, which we fix at the same
time.
And remove a spurious RESFILE in widget/windows/moz.build, where no binary
is produced, which means RESFILE had no meaning.
Differential Revision: https://phabricator.services.mozilla.com/D86154
Because while the original perl script was added to add version info
to Windows binaries, it does more and will do even more with upcoming
changes.
Differential Revision: https://phabricator.services.mozilla.com/D86153
It was only ever set to the same value as its default, except in
comm-central, where it is unset, but in directories that now don't link
anything (they did back when binary components were a thing).
Differential Revision: https://phabricator.services.mozilla.com/D85381
On Windows, with the double quotes, the command goes through the msys
shell, which then alters the command line such that empty arguments are
transformed into \ when they reach python.
With the single quotes, make invokes the command directly, without going
through the msys shell, and empty arguments are preserved.
Differential Revision: https://phabricator.services.mozilla.com/D86097
This is not a feature-for-feature rewrite. The python version removes
unused things, and simplifies some others:
- Only two command line arguments are taken in, and all the others are
dropped and the corresponding values are gotten from the buildconfig
module instead. The command line arguments are also taken as
positional arguments rather than going with a full argument parser.
- Variable expansion in module.ver used to be limited to one specific
variable to expand for a given value, which is now replaced with the
possibility to expand any of the variables that are allowed in
module.ver.
- The perl version was adding a RT_MANIFEST entry on its own if a
manifest file existed in the objdir for the given binary, but if such
a file existed, the build would fail after linking from the changes in
bug 1613799.
- The perl version was defaulting the module name to the binary name in
a branch that was never taken because the module name was assigned to
an empty string before that.
The output from the new script has been validated to being identical to
the output from the perl script, except for one extra whitespace at the
end of a comment.
Differential Revision: https://phabricator.services.mozilla.com/D85817
As the commit title suggests, this is a manual update, but we need to also
update `config/rules.mk` for the changed location of the wasi bindings.
Depends on D85408
Differential Revision: https://phabricator.services.mozilla.com/D85409
run-and-prefix.py is used as a wrapper of MAKE in moz-automation.mk, so
that the output is more clearly attributed to what part of the build is
happening, for everything that is post-build (e.g. package, upload,
etc.)
Ever since the switch to python 3, when it re-invokes MAKE, the
jobserver file descriptors would be closed and the sub-make wouldn't be
able to use it.
Differential Revision: https://phabricator.services.mozilla.com/D85051
It turns out setting CARGO_PROFILE_RELEASE_LTO has unwanted side
effects.
First it's not actually strictly equivalent to using `cargo rustc --
-Clto`. For instance, it apparently also enables cross-language LTO in
newer versions of cargo.
Second, it changes the rust computed hash for all the dependencies of
the crate being built with the variable set, which makes them diverge
from when the same dependencies are built through another crate in the
tree that is not LTOed. This effectively makes us build a _lot_ of
crates twice, many of which are not cacheable.
Since the original problem is that cargo >= 1.45 passes extra flags (`-C
embed-bitcode=no`) to rustc that are incompatible with `-Clto`, and while
it knows to adjust based on the `lto` setting in the build profile
(which CARGO_PROFILE_RELEASE_LTO overrides the default of), cargo
ignores flags passed via `cargo rustc -- ...` when making those
adjustments.
So, we need to override with `-C embed-bitcode=yes` on our own at the
same time we pass `-Clto`. But doing that through `cargo rustc -- ...`
is not enough because all the dependencies of the crate built with
`-Clto` need to be built with `-C embed-bitcode=yes`. So we need to
override with `RUSTFLAGS`, which will affect all the dependencies.
But we also need to do this consistently across all crates, not only the
dependencies of crates built with `-Clto`, otherwise we'd still end up
building crates twice (once with and once without the override).
Unfortunately, the `-C embed-bitcode=*` flag is also not supported in
versions older than 1.45, so we have to avoid adding it on older
versions.
We unfortunately support a large range of versions of rustc (albeit only
for tools/crashreporter), but we actually need to upgrade the smaller
supported version because rustc < 1.38 doesn't support our top-level
Cargo.lock. This makes the version check slightly less awful.
Differential Revision: https://phabricator.services.mozilla.com/D84652
The main reason we look at the complete symbols table is that before bug
1541792, we needed to look at that table for _NSModule symbols.
In bug 1516228, we also made everything llvm-objdump to limit the
differences cross-platform, but that's not necessary anymore per the
previous change.
llvm-objdump doesn't support getting only the dynamic symbols table, so
we go back to what we were using before bug 1516228, namely readelf,
while preserving a code path to use the complete symbols table for the
networking test on libgkrust.a, which doesn't have a dynamic symbols
table.
With this change, check_binary goes from 45s to 0.2s on my machine.
Differential Revision: https://phabricator.services.mozilla.com/D84305
The rules we create in the backend for generated files use a stub file
for the real file generation, and we do need dependencies on that
specific target for the changes in bug 1264836 to work without races.
Differential Revision: https://phabricator.services.mozilla.com/D84311
Otherwise, crates that are shared between host and target are always
rebuilt because of the difference in rust flags.
Differential Revision: https://phabricator.services.mozilla.com/D83412
Doing this means that make will correctly understand that the underlying
program has changed and therefore it needs to trigger install rules.
Differential Revision: https://phabricator.services.mozilla.com/D83326
This makes dir/target, dir/target-objects, etc. depend on
dir/pre-compile, which ensures all items in the pre-compile pseudo-tier
are built before entering in the directory they are defined in, during
the compile tier. This removes the need for EXTRA_DEPS hacks.
Differential Revision: https://phabricator.services.mozilla.com/D83036
Generated files marked as "required during compile" are limited to a
number of extensions, most of which are source file types, so they get
generated when compiling the corresponding object file. Other types,
though, are currently handled via EXTRA_DEPS, which is clunky.
As of now, these other types are limited to link-related items, but more
would be useful (e.g. .inc files, which are only included in one
location).
This however works against the static analysis tasks (coverity and
clang-tidy), which currently rely on everything they need being created
via the export tier. That excludes using EXTRA_DEPS-based hacks.
We create a "pre-compile" tier, that is not used during a normal build,
but can be invoked manually, which the static analysis tasks will do.
Differential Revision: https://phabricator.services.mozilla.com/D83035
The previous logic would check the value of the `PYTHON3` environment variable, which is not set by the Firefox build system, so this build script tends to just use the `python3` at the tip of the `PATH`, regardless of which value you've configured for it. Instead, it should prefer to read from the `OBJDIR` if one is present.
Differential Revision: https://phabricator.services.mozilla.com/D82981
This makes it much simpler for the for the static JS services cache to store
and lookup interface IDs for components.
Differential Revision: https://phabricator.services.mozilla.com/D81416
... instead of a C file. This avoids issues with sccache (via dist or
caching), because sccache doesn't do anything with assembly files,
but with a C file, it doesn't know about the incbin-included data file.
This has the unfortunate side effect of requiring some manual work to
mark the file for safeseh on 32-bits Windows.
Differential Revision: https://phabricator.services.mozilla.com/D82814
This creates a general pytest plugin under "config/mozunit" which implements
the ability to mark tests as "slow". Slow tests will be skipped by default
unless "--run-slow" is passed in.
Differential Revision: https://phabricator.services.mozilla.com/D81402
This makes it much simpler for the for the static JS services cache to store
and lookup interface IDs for components.
Differential Revision: https://phabricator.services.mozilla.com/D81416
This is strictly a quality of life improvement when cross-compiling to
Windows targets. A common scenario is mounting an object directory
into a Windows VM, which requires additional VM configuration to
handle (absolute) symlinks. With this patch, `export NSDISTMODE=copy`
in a mozconfig sidesteps any such symlink issues.
Differential Revision: https://phabricator.services.mozilla.com/D77118
For a while, now, ICU data has been stored as raw read-only data in rather
than a separate loadable library. This means the parts of ICU that can load
such a module are not necessary.
Differential Revision: https://phabricator.services.mozilla.com/D82262
All the supported compilers support a GNU AS-like syntax, with only a
few details varying. It means we can use a single, simpler, way to
include the ICU data file, instead of 3 different ways, including one
that uses armasm64.exe, possibly wrapped with Wine.
Differential Revision: https://phabricator.services.mozilla.com/D82144
Because jar_maker is not in the libs tier, we also rename the libs-%
targets for l10n repacks to l10n-%, which make it clearer what they are
for.
And because multilocale.txt is both a GeneratedFile and a file that is
generated (and installed) via manual build rules, keeping it in the misc
target actually breaks building in toolkit/locales during l10n repacks,
so move it to libs for now.
Differential Revision: https://phabricator.services.mozilla.com/D81766
While ideally we'd just move all of them at the top-level, there are
other things that depend on them that wouldn't then get the right
dependencies. One of them is install steps in dist/something, but there
are others, and that's a rather long pole of things requiring many
changes along with chances to break things.
So instead, we go with a simpler and more limited approach, where we
still recurse into directories to run their export tier (to run whatever
else than GeneratedFiles they have), but ensure the GeneratedFiles we
moved at the top-level are built before recursing by making
directory/export depend on them.
Differential Revision: https://phabricator.services.mozilla.com/D80608
While ideally we'd just move all of them at the top-level, there are
other things that depend on them that wouldn't then get the right
dependencies. One of them is install steps in dist/something, but there
are others, and that's a rather long pole of things requiring many
changes along with chances to break things.
So instead, we go with a simpler and more limited approach, where we
still recurse into directories to run their export tier (to run whatever
else than GeneratedFiles they have), but ensure the GeneratedFiles we
moved at the top-level are built before recursing by making
directory/export depend on them.
Differential Revision: https://phabricator.services.mozilla.com/D80608
While ideally we'd just move all of them at the top-level, there are
other things that depend on them that wouldn't then get the right
dependencies. One of them is install steps in dist/something, but there
are others, and that's a rather long pole of things requiring many
changes along with chances to break things.
So instead, we go with a simpler and more limited approach, where we
still recurse into directories to run their export tier (to run whatever
else than GeneratedFiles they have), but ensure the GeneratedFiles we
moved at the top-level are built before recursing by making
directory/export depend on them.
Differential Revision: https://phabricator.services.mozilla.com/D80608
They were necessary some time ago, but:
- there is no js/xpconnect/src/export target anymore,
- accessibility/xpcom/export had a dependency on xpcom/xpidl/export for
the ply cache files, but they're not used as of bug 1633156.
- the recursive make backend was adding automatic dependencies on
xpcom/xpidl/export for the ply cache files.
Differential Revision: https://phabricator.services.mozilla.com/D80150
Replace the duplicate lists in mozglue/baseprofiler/public and js/public with
a shared list. Add this list to both moz.build files so it is published twice
which simplifies supporting different standalone configurations.
Differential Revision: https://phabricator.services.mozilla.com/D79520
IMPACTED_TESTS was added in bug 1184405, but is under-utilized in current moz.build
annotations; also bug 1516847 removed use of IMPACTED_TESTS in 'mach test'. This
patch removes remaining support for IMPACTED_TESTS.
Differential Revision: https://phabricator.services.mozilla.com/D79141
Extremely common file paths on macOS like `/Users/...` are interpreted
as `/U...` flags by `clang-cl`. This is so common that there's a
`-Wslash-u-filename` warning. Ensure that file paths are considered
paths when compiling by terminating options with `--`.
This commit handles everything except assembler invocations, because
at least `nasm` doesn't handle `--`.
Differential Revision: https://phabricator.services.mozilla.com/D77119
This is strictly a quality of life improvement when cross-compiling to
Windows targets. A common scenario is mounting an object directory
into a Windows VM, which requires additional VM configuration to
handle (absolute) symlinks. With this patch, `export NSDISTMODE=copy`
in a mozconfig sidesteps any such symlink issues.
Differential Revision: https://phabricator.services.mozilla.com/D77118
Mike Hommey
Cosmin Sabou
Narcis Beleuzu
Mozilla Releng Treescript
Paulo Matias
Ricky Stewart
Razvan Maries
Jed Davis
Christian Holler
Nico Grunbaum
Nathan Froyd
Bogdan Tara
Mihai Alexandru Michis
Jeff Walden
Butkovits Atila
Tom Prince
Kris Maglione
Csoregi Natalia
Andrew Halberstadt
Axel Hecht
Nick Alexander
André Bargull
Emilio Cobos Álvarez
Coroiu Cristina
Dorel Luca
Ted Campbell
Geoff Brown