While this isn't related to the bug, since we're going to touch the
cargo compiler flags, we might as well do this too.
It wasn't previously reliable to pass those flags down because what
cargo uses as target for build scripts and procedural macros is
determined by the rust host, which was not necessarily the same as the
build system host. But as of bug 1523143, they are always the same.
Differential Revision: https://phabricator.services.mozilla.com/D18280
--HG--
extra : moz-landing-system : lando
Now that Make invokes cargo without going through an msys shell,
environment variables are going to be preserved properly, and we can now
"safely" pass the compiler-related variables down to cargo on Windows.
This makes rust target builds use the expected compiler and flags,
instead of the cc-rs crate guessing, picking cl.exe, and using the wrong
one, with the build later failing when linking it all together because
one of the objects is not for the right target.
Interestingly, the lmdb code is today built for the wrong target on
aarch64, but somehow, it doesn't break the build on automation,
presumably because the lmdb code is actually dead code, and the linker
eliminates the object as unused, masking the problem.
Depends on D18186
Differential Revision: https://phabricator.services.mozilla.com/D18187
--HG--
extra : moz-landing-system : lando
Double quotes on a command line forces Make to use a msys shell when
invoking the command. Single quotes don't have this effect. This is the
last bit that prevented Make from invoking cargo directly on Windows.
Depends on D18184
Differential Revision: https://phabricator.services.mozilla.com/D18185
--HG--
extra : moz-landing-system : lando
These require some awkward setup to keep things working on
non-cross-compiles on non-Windows, but we'll change that shortly in a
later bug.
Depends on D18183
Differential Revision: https://phabricator.services.mozilla.com/D18184
--HG--
extra : moz-landing-system : lando
This is a drive-by change, allowing to keep the
force-cargo-library-build recipe more like the others.
Depends on D18181
Differential Revision: https://phabricator.services.mozilla.com/D18182
--HG--
extra : moz-landing-system : lando
The `env` program, on windows, comes from msys, so invoking `env cargo`
guarantees an msys roundtrip, which usually breaks environment variable
in interesting ways.
This moves most of the environment variables we set with `env` (the
easiest ones) to exporting the same values from make itself.
Depends on D18180
Differential Revision: https://phabricator.services.mozilla.com/D18181
--HG--
extra : moz-landing-system : lando
Because we're going to change how cargo recipes are called to export
environment variables rather than by wrapping the call with `env`, to
avoid msys roundtrips, it's better to avoid the complexity when not
building rust, and including a separate file only when required helps
with that. It is also possible to wrap the entire rust section of
rules.mk in the same condition we use for the include, but using a
separate file also makes things clearer.
Differential Revision: https://phabricator.services.mozilla.com/D18180
--HG--
rename : config/rules.mk => config/makefiles/rust.mk
extra : moz-landing-system : lando
The build system has skipped creating target static libraries for very
long, except in very specific cases.
We can actually do the same for host static libraries, for which we
don't even need the escape hatch to still allow to create static
libraries.
Depends on D15171
Differential Revision: https://phabricator.services.mozilla.com/D15172
--HG--
extra : moz-landing-system : lando
Summary:
This patch ports xptcodegen.py over to the new perfecthash.py system, removing
some special-case code generators, and taking advantage of the easier-to-use
interface.
In addition, the code was changed to take advantage of the endianness
information from Part 2, allowing us to avoid having to perform endianness swaps
at runtime when hashing nsIDs.
Depends On D2616
Reviewers: froydnj!
Tags: #secure-revision
Bug #: 1479484
Differential Revision: https://phabricator.services.mozilla.com/D2618
The build system knows at build-backend time where to find each IDL
file; making xpidl-process.py rediscover this by requiring
xpidl-process.py to search through directories to find input IDL files
is silly. To rememdy this, we're going to modify things so full paths
are passed into the script. Those paths can then be used directly, with
no searching.
The tail end of the xpidl Makefile.in contains a line, generated for
every xpt file:
$(1): $(addsuffix .idl,$(addprefix $(dist_idl_dir)/,$($(basename $(notdir $(1)))_deps)))
This line, in context, is saying that the xpt file depends on all of its
input IDL files. But xpidl-process.py already generates this
information when we pass it --depsdir, which we do. So this code is
redundant with what we already generate, and it can be removed.
The previous patch required us to pass a single -I argument pointing at
$(DIST)/idl so IDL include statements would work correctly. This patch
lifts that limitation and explicitly points xpidl-process.py at the
locations of all the IDL source directories to search for included IDL
files. Invocations of xpidl-process.py no longer depend on IDL files
being copied to the objdir.
Building on the last patch, we can change the build process to pass in
the directories where the input IDL files can be found. It is
convenient to pass in just the relative source directory paths, to
encourage people to not look in the object directory and to make the
command lines slightly shorter.
xpidl-process.py still assumes that included IDL files can be found by
looking in a single directory. We add a single -I argument to the
invocation of xpidl-process.py to accommodate this short-sightedness.
The current IDL build setup assumes that all IDL files can be found in a
single directory. This setup requires that all IDL files be copied to a
single directory, which is suboptimal in terms of disk I/O and also
complicates things like generating IDL files at build time.
As a first step in moving away from this state of affairs,
xpidl-process.py needs to be taught that the input IDL files could
potentially be found in multiple directories. The current setup can
just specify $(DIST)/idl as the lone directory to examine. Future
patches will change this to examine multiple directories.
This patch contains the meat of the changes here. The following summarize the changes:
1. xptinfo.h is rewritten to expose the new interface for reading the XPT data,
The nsXPTInterfaceInfo object exposes methods with the same signatures as
the methods on nsIInterfaceInfo, to make converting code which used
nsIInterfaceInfo as easy as possible, even when those methods don't have
signatures which make a ton of sense anymore. There are also a few methods
which are unnecessary (they return `true` or similar), which should be
removed over time.
Members of the data structures are made private in order to prevent reading
them directly. Code should instead call the getter methods. This should make
it easier to change their memory representation in the future. Constructing
these structs is made possible by making the structs `friend class` with the
XPTConstruct class, which is implemented by the code generator, and is able
to access the private fields.
In addition, rather than using integers with flag constants, I opted for
using C++ bitfields to store individual flags, as I found it made it easier
to both write the code generator, and reason about the layouts of the types.
I was able to shave a byte off of each nsXPTParamInfo (4 bytes -> 3 bytes)
by shoving the flags into spare bits in the nsXPTType. Unfortunately there
was not enough room for the retval flag. Fortunately, we already depend in
our code on the retval parameter being the last parameter, so I worked
around this by removing the retval flag and instead having a `hasretval`
flag on the method itself.
2. An xptinfo.cpp file is added for out-of-line definitions of more complex
methods, and the internal implementation details of the perfect hash.
Notable is the handling of xptshim interfaces. As the type is uniform, a
flag is checked when trying to read constant information, and a different
table with pointers into webidl data structures is checked when the type is
determined to be a shim.
Ideally we could remove this once we remove the remaining consumers of the
existing shim interfaces.
3. A python code generator which takes in the json XPT files generated in the
previous part, and emits a xptdata.cpp file with the data structures. I did
my best to heavily comment the code.
This code uses the friend class trick to construct the private fields of the
structs, and avoid a dependency on the ordering of fields in xptinfo.h.
The sInterfaces array's order is determined by a generated perfect hash
which is also written into the binary. This should allow for fast lookups by
IID or name of interfaces in memory. The hash function used for the perfect
hash is a simple FNV hash, as they're pretty fast.
For perfect hashing of names, another table is created which contains
indexes into the sInterfaces table. Lookup by name is less common, and this
form of lookup should still be very fast.
4. The necessary Makefiles are updated to use the new code generator, and
generate the file correctly.
This patch handles the actual generation of the static data structures
used to represent XPT information. XPT files are generated in the same
way as they are now, but they are used only as an intermediate
representation to speed up incremental compilation rather than
something used by Firefox itself. Instead of linking XPTs into a
single big XPT file at packaging time, they are linked into a single
big C++ file at build time, that defines the various static consts in
XPTHeader.
In xpt.py, every data structure that can get written to disk gets an
additional code_gen() method that returns a representation of that
data structure as C++ source code. CodeGenData aggregates this
information together, handling deduplication and the final source code
generation.
The ctors are needed for XPTConstValue to statically initialize the
different union cases without resorting to designated initializers,
which are part of C99, not C++. Designated initializers appear to be
supported in C++ code by Clang and GCC, but not MSVC. The ctors must
be constexpr to ensure they are actually statically initialized so
they can be shared between Firefox processes.
I also removed an unnecessary "union" in XPTConstDescriptor.
Together, these patches reduce the amount of memory reported by
xpti-working-set from about 860,000 bytes to about 200,000 bytes. The
remaining memory is used for xptiInterface and xptiTypelibGuts (which
are thin wrappers around the XPT interfaces and header) and hash
tables to speed up looking up interfaces by name or IID. That could
potentially be eliminated from dynamic allocations in follow up
work. These patches did not affect memory reporting because XPT arenas
are still used by the remaining XPTI data structures.
MozReview-Commit-ID: Jvi9ByCPa6H
--HG--
extra : rebase_source : a9e48e7026aab4ad1b7f97e50424adf4e3f4142f
Now that XPT files are not loaded from files at runtime, code for
packaging XPT files can be removed.
This means that a couple of test XPIDL interfaces will get shipped in
builds to users that weren't before, but I don't think that matters
much.
This also puts XPT files into the local objdir for the XPIDL makefile,
instead of dist/bin, because they are no longer part of the
distribution.
MozReview-Commit-ID: 7gWj8KWUun3
--HG--
extra : rebase_source : 65bac47c2cd1a20b3c675a01b44a25a1d2d3ab7a
This patch handles the actual generation of the static data structures
used to represent XPT information. XPT files are generated in the same
way as they are now, but they are used only as an intermediate
representation to speed up incremental compilation rather than
something used by Firefox itself. Instead of linking XPTs into a
single big XPT file at packaging time, they are linked into a single
big C++ file at build time, that defines the various static consts in
XPTHeader.
In xpt.py, every data structure that can get written to disk gets an
additional code_gen() method that returns a representation of that
data structure as C++ source code. CodeGenData aggregates this
information together, handling deduplication and the final source code
generation.
The ctors are needed for XPTConstValue to statically initialize the
different union cases without resorting to designated initializers,
which are part of C99, not C++. Designated initializers appear to be
supported in C++ code by Clang and GCC, but not MSVC. The ctors must
be constexpr to ensure they are actually statically initialized so
they can be shared between Firefox processes.
I also removed an unnecessary "union" in XPTConstDescriptor.
Together, these patches reduce the amount of memory reported by
xpti-working-set from about 860,000 bytes to about 200,000 bytes. The
remaining memory is used for xptiInterface and xptiTypelibGuts (which
are thin wrappers around the XPT interfaces and header) and hash
tables to speed up looking up interfaces by name or IID. That could
potentially be eliminated from dynamic allocations in follow up
work. These patches did not affect memory reporting because XPT arenas
are still used by the remaining XPTI data structures.
MozReview-Commit-ID: Jvi9ByCPa6H
--HG--
extra : rebase_source : 719dfbcb9f83235c0f1f0766270b7f127f9ab04e
Now that XPT files are not loaded from files at runtime, code for
packaging XPT files can be removed.
This means that a couple of test XPIDL interfaces will get shipped in
builds to users that weren't before, but I don't think that matters
much.
This also puts XPT files into the local objdir for the XPIDL makefile,
instead of dist/bin, because they are no longer part of the
distribution.
MozReview-Commit-ID: 7gWj8KWUun3
--HG--
extra : rebase_source : 6f7d4fd1d6cdea2c14866705a2dc972eb5f43382
Historically we built all our binaries in directories in the objdir, then
symlinked them into dist/bin. Some binaries needed to be copied instead
so that certain relative path lookups work properly, so we resorted to
sprinkling `NSDISTMODE=copy` around Makefiles.
This change makes it so we build PROGRAMs (not any other sort of targets)
directly in dist/bin instead. We could do the same for our other targets
with a little more work.
There were several places in the tree that were copying built binaries to
some other place and needed fixup to match the new location of binaries.
On Windows pdb files are left in the objdir where the program was
originally linked. symbolstore.py needs to locate the pdb file both to
determine whether it should dump symbols for a binary and also to copy
the pdb file into the symbol package. We fix this by simply looking for
the pdb file in the current working directory if it isn't present next
to the binary, which matches how we invoke symbolstore.py.
MozReview-Commit-ID: 8TOD1uTXD5e
The last APK produced using the ANDROID_APK_* moz.build/Makefile.in
mechanism was Robocop, so we can get rid of these now.
MozReview-Commit-ID: 9b08ZvvOAoC
--HG--
extra : rebase_source : ac4fea057bf6e731b0f26a1b6902f17a7362076d
Final target Rust programs are currently not picked up and
symlinked/copied to ${objdir}/dist/bin because the RUST_PROGRAMS output
variable is missing from the if-condition.
This change causes Rust binaries from RUST_PROGRAMS to be placed alongside
the other final target programs.
MozReview-Commit-ID: 5OIH1UMmCq2
--HG--
extra : rebase_source : cfcff21e9371b920c895ad27df344008962d7471
rustc generates .lib files for its libraries when compiling for Windows
(even using MinGW on Linux). But MinGW expects .a files. So we add in
rust-specific prefix and suffixes so MinGW builds can find the libs that
rustc generates. (And the RUST_LIB- variables default to the same vales
as the LIB_ variables otherwise.)
MozReview-Commit-ID: ClsA0YuJaxh
--HG--
extra : rebase_source : 7b46460c94ceb34b7a5a302ce91d3f1dca600041
By using $(REPORT_BUILD) instead of just echoing the filename, there is
no change during a regular build without REBUILD_CHECK, but specifying
REBUILD_CHECK in the environment will show which files triggered an .xpt
to rebuild. This is helpful in debugging why these files may be built
unnecessarily.
MozReview-Commit-ID: GGNaKAl02Ea
--HG--
extra : rebase_source : 098a81265deed9afc0b284d9863f18ebd74fda33
The only complicating factor here is having to split out the --target
flag from cargo_build_flags, so we can pass the appropriate one
depending on our build target.
This removes the unnecessary setting of c-basic-offset from all
python-mode files.
This was automatically generated using
perl -pi -e 's/; *c-basic-offset: *[0-9]+//'
... on the affected files.
The bulk of these files are moz.build files but there a few others as
well.
MozReview-Commit-ID: 2pPf3DEiZqx
--HG--
extra : rebase_source : 0a7dcac80b924174a2c429b093791148ea6ac204
By bug 1270621, ETW support is removed. So no one uses messege compiler.
MozReview-Commit-ID: HGUAkrb208N
--HG--
extra : rebase_source : 5178283b781e4805b4e97f09b35a4150119d060a
extra : histedit_source : 40d6992894315b51863db4e7697df1c1ef18edd8
We can just generate xpidllex.py/xpidlyacc.py in the current directory
rather than one directory higher, and specify this directory as an
include path to xpidl-process.py
MozReview-Commit-ID: KLoGjudc4Y8
--HG--
extra : rebase_source : 8dda268c6490cdfb8b896de9da5b789208584193
The behavior is not entirely idempotent (most notably for
buildconfig.html), but this can be improved later if necessary.
It is idempotent where it matters.
This allows to get rid of config/makefiles/rcs.mk and its uses.
This might seem like going in the opposite direction of what we tend to do
to move to moz.build land, but those flags are irrelevant in many situations
and are better separated out.
This new ChromeManifestEntry object type is generic and can hold any kind of
chrome manifest entry, but we currently only emit them for binary components.
References to sub-directory manifests is left to the backend, for now, until
all manifest entries are emitted by the frontend.
Mike Hommey
Chris Manchester
Narcis Beleuzu
Nika Layzell
Nathan Froyd
Nick Alexander
Andrew McCreight
Dorel Luca
Ted Mielczarek
arthur.iakab
Kim Moir
Andreas Tolfsen
Tom Ritter
Mike Shal
Tom Tromey
Makoto Kato