This will make sure that when running |mach python-test --python 3| locally,
we only run the tests that also run in CI with python 3 (and therefore pass
presumably).
MozReview-Commit-ID: 3OBr9yLSlSq
--HG--
extra : rebase_source : 456340d0ecdddf1078f2b5b4ebb1eddf3813b26a
Unfortunately, I wasn't able to figure out a way to make firefox build & run in
the intermediate stages of these commits. Because of this, I am going to just
delete most of the code which I am deleting in the first patch, as I figure that
those are somewhat uninteresting changes, and then make the other changes in the
following patches.
In total, the following things are deleted:
1. All of xpcom/typelib, except for `xpt/tools` - this directory is being
subsumed entirely into xpcom/reflect/xptinfo.
2. Most of the code in xpcom/reflect/xptinfo, it is being rewritten to avoid
allocating and contain all of the necessary data structures.
3. idl-parser's typelib.py XPT generator, as it will be replaced.
4. Most includes of files which have been deleted.
NOTE: xpcom/typelib/xpt/tools/xpt.py was not removed, as it is used by bundling
code & bundling tests, which we don't want to remove yet.
Many of these fields have accessors, and can only be read indirectly
by going through the XPTHeader data structure anyways, so they should
be marked private. This makes the generated XPT data file noisier due
to the need for constexpr constructors.
I had to fix the ctors for the classes in xptinfo.h to be less weird
because there was a compiler error.
Members in two of the classes need to be marked protected because they
have subclasses in xptinfo.h. Ideally those classes would be merged
in.
MozReview-Commit-ID: 70IdFAhp5je
--HG--
extra : rebase_source : 48cb1ec84c872b8fd6bb8c07f8ad2b47d0173f28
Many of these fields have accessors, and can only be read indirectly
by going through the XPTHeader data structure anyways, so they should
be marked private. This makes the generated XPT data file noisier due
to the need for constexpr constructors.
I had to fix the ctors for the classes in xptinfo.h to be less weird
because there was a compiler error.
Members in two of the classes need to be marked protected because they
have subclasses in xptinfo.h. Ideally those classes would be merged
in.
MozReview-Commit-ID: 70IdFAhp5je
--HG--
extra : rebase_source : a2670a65ace291defc47845c4058477f0ae5360a
This patch handles the actual generation of the static data structures
used to represent XPT information. XPT files are generated in the same
way as they are now, but they are used only as an intermediate
representation to speed up incremental compilation rather than
something used by Firefox itself. Instead of linking XPTs into a
single big XPT file at packaging time, they are linked into a single
big C++ file at build time, that defines the various static consts in
XPTHeader.
In xpt.py, every data structure that can get written to disk gets an
additional code_gen() method that returns a representation of that
data structure as C++ source code. CodeGenData aggregates this
information together, handling deduplication and the final source code
generation.
The ctors are needed for XPTConstValue to statically initialize the
different union cases without resorting to designated initializers,
which are part of C99, not C++. Designated initializers appear to be
supported in C++ code by Clang and GCC, but not MSVC. The ctors must
be constexpr to ensure they are actually statically initialized so
they can be shared between Firefox processes.
I also removed an unnecessary "union" in XPTConstDescriptor.
Together, these patches reduce the amount of memory reported by
xpti-working-set from about 860,000 bytes to about 200,000 bytes. The
remaining memory is used for xptiInterface and xptiTypelibGuts (which
are thin wrappers around the XPT interfaces and header) and hash
tables to speed up looking up interfaces by name or IID. That could
potentially be eliminated from dynamic allocations in follow up
work. These patches did not affect memory reporting because XPT arenas
are still used by the remaining XPTI data structures.
MozReview-Commit-ID: Jvi9ByCPa6H
--HG--
extra : rebase_source : a9e48e7026aab4ad1b7f97e50424adf4e3f4142f
With fully linked XPT data, there is exactly one directory entry per
descriptor, plus one per unresolved interface. There are 1200 of the
former and 40 of the latter. By merging them, we save a 32 bit int per
directory entry, at the cost of 11 bytes per unresolved interface.
This will make VerifyAndAddEntryIfNew slightly slower because it has
to do an nsID equality check rather than a null check, but I can't
imagine that will matter.
My main goal for this patch is to reduce the size of the executable,
to avoid a regression with my static XPT information patches, but it
should reduce memory a little bit, too.
MozReview-Commit-ID: L35YdOuAyF4
--HG--
extra : rebase_source : 8358a73a0f8f81081661538d4a7c9a31b1aa7a56
To allow XPT information to be shared between processes, it needs to
not contain any pointers, because they cause relocations. I've
eliminated pointers by hoisting all of the variable length data
structures to XPTHeader, into a single array for each type of
data. These data structures now use an index into these arrays to find
their first element. Strings are similar, but they are mashed into a
single giant string, including embedded null terminators. Modifying
the accessor methods to support this is easy, because there is only a
single global instance of each of these arrays in XPTHeader.
MozReview-Commit-ID: 5rgwaEBvDYl
--HG--
extra : rebase_source : 2e423f088d662920a89f3b66c70d26fe340b5fce
Now that there is only one XPTHeader, we can devolve the fields in it
to avoid some indirection. The biggest part here is getting rid of the
mHeader field on xptiTypelibGuts.
The array is [] instead of * to avoid a relocation, by ensuring that
XPTHeader::kInterfaceDirectory as well as the data it points to cannot
be changed.
MozReview-Commit-ID: AzvNTNZKkfi
--HG--
extra : rebase_source : d911a54b3db1f9f57b538d13ae86f28965ab33b3
This patch removes C++ code related to reading in XPT information from
files. (Code related to packaging XPT files will be removed in the
next patch.) This includes code in the manifest parser, in addition to
the actual code for parsing files.
XPT information is now loaded directly from a single static data
structure, XPTHeader::kHeader, which will be automatically generated
at compile time from .idl files (via .xpt files). Note that the script
to do that is not added until part 6 of this patch series, so linking
will fail on parts 2 through 5.
I inlined XPTInterfaceInfoManager::RegisterXPTHeader into the ctor,
because that is the only caller. It feels like the lock there should
not be needed any more, but I left it alone for now.
The forward declaration of XPTArena in xptiprivate.h is needed because
it was being bootlegged via xpt_xdr.h. Some of the data structures in
reflect/xptinfo/ (which wrap the xpt_struct.h data structures) are
still allocated using XPTArena. Hopefully we can get rid of that in
followup work.
I also deleted a lot of comments in xpt_struct.h that talk about the
on-disk format. I also deleted checking of the major version number,
because that should not matter when the XPT information is baked into
the executable.
MozReview-Commit-ID: 6NJdaCWRBhU
--HG--
extra : rebase_source : 6512a05f2a8bee1e6e6b0423d2cb376d8c34728b
This lets us hide later changes to how these arrays are stored. There
should be no behavioral change. Some methods in xpt_struct.h are
declared inline at the end of the header due to the order that classes
are declared in the header.
MozReview-Commit-ID: KAxUKn3sDOD
--HG--
extra : rebase_source : 867ce100e5178c85485e6c3bac5d6bfca098f78b
This patch handles the actual generation of the static data structures
used to represent XPT information. XPT files are generated in the same
way as they are now, but they are used only as an intermediate
representation to speed up incremental compilation rather than
something used by Firefox itself. Instead of linking XPTs into a
single big XPT file at packaging time, they are linked into a single
big C++ file at build time, that defines the various static consts in
XPTHeader.
In xpt.py, every data structure that can get written to disk gets an
additional code_gen() method that returns a representation of that
data structure as C++ source code. CodeGenData aggregates this
information together, handling deduplication and the final source code
generation.
The ctors are needed for XPTConstValue to statically initialize the
different union cases without resorting to designated initializers,
which are part of C99, not C++. Designated initializers appear to be
supported in C++ code by Clang and GCC, but not MSVC. The ctors must
be constexpr to ensure they are actually statically initialized so
they can be shared between Firefox processes.
I also removed an unnecessary "union" in XPTConstDescriptor.
Together, these patches reduce the amount of memory reported by
xpti-working-set from about 860,000 bytes to about 200,000 bytes. The
remaining memory is used for xptiInterface and xptiTypelibGuts (which
are thin wrappers around the XPT interfaces and header) and hash
tables to speed up looking up interfaces by name or IID. That could
potentially be eliminated from dynamic allocations in follow up
work. These patches did not affect memory reporting because XPT arenas
are still used by the remaining XPTI data structures.
MozReview-Commit-ID: Jvi9ByCPa6H
--HG--
extra : rebase_source : 719dfbcb9f83235c0f1f0766270b7f127f9ab04e
With fully linked XPT data, there is exactly one directory entry per
descriptor, plus one per unresolved interface. There are 1200 of the
former and 40 of the latter. By merging them, we save a 32 bit int per
directory entry, at the cost of 11 bytes per unresolved interface.
This will make VerifyAndAddEntryIfNew slightly slower because it has
to do an nsID equality check rather than a null check, but I can't
imagine that will matter.
My main goal for this patch is to reduce the size of the executable,
to avoid a regression with my static XPT information patches, but it
should reduce memory a little bit, too.
MozReview-Commit-ID: L35YdOuAyF4
--HG--
extra : rebase_source : b3d0780808f9fc708b5b2dffeda45030974dc0bf
To allow XPT information to be shared between processes, it needs to
not contain any pointers, because they cause relocations. I've
eliminated pointers by hoisting all of the variable length data
structures to XPTHeader, into a single array for each type of
data. These data structures now use an index into these arrays to find
their first element. Strings are similar, but they are mashed into a
single giant string, including embedded null terminators. Modifying
the accessor methods to support this is easy, because there is only a
single global instance of each of these arrays in XPTHeader.
MozReview-Commit-ID: 5rgwaEBvDYl
--HG--
extra : rebase_source : 6c35fd1034b727443bb87dfde0a4af4799ed480a
Now that there is only one XPTHeader, we can devolve the fields in it
to avoid some indirection. The biggest part here is getting rid of the
mHeader field on xptiTypelibGuts.
The array is [] instead of * to avoid a relocation, by ensuring that
XPTHeader::kInterfaceDirectory as well as the data it points to cannot
be changed.
MozReview-Commit-ID: AzvNTNZKkfi
--HG--
extra : rebase_source : 7b2043cecee5e8d063ea4d0b4a2223b2261c62e7
This patch removes C++ code related to reading in XPT information from
files. (Code related to packaging XPT files will be removed in the
next patch.) This includes code in the manifest parser, in addition to
the actual code for parsing files.
XPT information is now loaded directly from a single static data
structure, XPTHeader::kHeader, which will be automatically generated
at compile time from .idl files (via .xpt files). Note that the script
to do that is not added until part 6 of this patch series, so linking
will fail on parts 2 through 5.
I inlined XPTInterfaceInfoManager::RegisterXPTHeader into the ctor,
because that is the only caller. It feels like the lock there should
not be needed any more, but I left it alone for now.
The forward declaration of XPTArena in xptiprivate.h is needed because
it was being bootlegged via xpt_xdr.h. Some of the data structures in
reflect/xptinfo/ (which wrap the xpt_struct.h data structures) are
still allocated using XPTArena. Hopefully we can get rid of that in
followup work.
I also deleted a lot of comments in xpt_struct.h that talk about the
on-disk format. I also deleted checking of the major version number,
because that should not matter when the XPT information is baked into
the executable.
MozReview-Commit-ID: 6NJdaCWRBhU
--HG--
extra : rebase_source : e169b827a64bad5dde15de6be0b2ff5e7aa35e3f
This lets us hide later changes to how these arrays are stored. There
should be no behavioral change. Some methods in xpt_struct.h are
declared inline at the end of the header due to the order that classes
are declared in the header.
MozReview-Commit-ID: KAxUKn3sDOD
--HG--
extra : rebase_source : 782fddc7bbf29c8ec73a0c6dbcca0621739e6c6e
For bug 1438688, I need to statically initialize all of the data types
in xpt_struct.h. It turns out that the approach I was using for unions
is C99 and not C++. While for some reason Clang and GCC are okay with
that, MSVC is not.
One of the unions is a gnarly anonymous union in
XPTTypeDescriptor. However, every arm of this union is either 1 or 2
uint8_t, so I think it is reasonable to eliminate the union and
replace it with two fields, mData1 and mData2. I've fixed up the
places that read this data to use accessor methods with nice asserts
about the tag on the variant, so if anything I think that part is
nicer.
The code in xpt_struct.cpp that writes to this data is maybe less
nice, but I'm deleting it in bug 1438688 so I think that's okay. We
also lose some detail in the comments about the relationship between
what is stored in memory compared to what is stored on disk, but I
think that's okay, too.
MozReview-Commit-ID: DGi19f8HnMi
--HG--
extra : rebase_source : e3535ecb3d7f02bdb643df153e8aba3e106d9104
Most of the classes in xpt.py define a helper method to encode the
values of the flags into a byte, but for some reason Method does
not. It'll be useful to have one for Method for the C++ backend I am
working on.
MozReview-Commit-ID: ESi1CnstbN2
--HG--
extra : rebase_source : 23822e7770e9a3d1b5a2359ae9753f950af3a121
The trick here is that we read in the element type before we expand the array, so that we
can write it to the new array before it becomes constified.
MozReview-Commit-ID: 2pbpNVZ3gPZ
--HG--
extra : rebase_source : 4c07fff32ea6b1d5c60ae3e40fb869b737cab13e
Also, change some reinterpret casts to static casts, because there was
no need for them to be reinterpret.
MozReview-Commit-ID: EtPmwxboaq9
--HG--
extra : rebase_source : 59fe2d74c8567af4d8000cb230e8dc0f8bf728ff
A lot of time is spent during the final big XPT link determining what
the index is for each interface. Changing this to use a map
eliminates about 2/3 of the running time. This patch reduces the run
time to a little under a second on my local OSX machine.
MozReview-Commit-ID: CH4OYXtT19q
--HG--
extra : rebase_source : 6d6f755c57dcbf20112768583948f851b8bf34bf
This was added in bug 1081000 to support linking XPT tests, but those
tests were removed in bug 1248534, part 1, so this shouldn't be needed
any more.
MozReview-Commit-ID: I1V2XVBaMG7
--HG--
extra : rebase_source : 8e772da227782b5304b553c0ba85f96f708bea99
Also, get rid of a gratuitous use of a trinary operator in
nsXPCWrappedJSClass::CallMethod, clean up the style a little, and mark
an unimplemented ctor as deleted.
MozReview-Commit-ID: Kp64sMxyRWc
--HG--
extra : rebase_source : e6082003d3759234cd5f4630b5560b14930c0a88
Also, clean up the style a little and mark an unimplemented ctor as
deleted.
MozReview-Commit-ID: JqmveE6qWFa
--HG--
extra : rebase_source : 62c8249de1f52686b4dd5d2a043261d2618d7433
Also, remove a few unused things.
Removing the include of xpt_arena.h from xpt_struct.h required that I
added it to two other files.
MozReview-Commit-ID: 4bMDRYt0Zxc
--HG--
extra : rebase_source : 91548b62dbf4b92bf918d196067e6fabb9a72302
This makes the file 2-space indented, gets rid of padding between
types and members, moves the * to the left, fixes the mode line,
license and include guards, and fixes up the first line of some of the
multiline comments. I also reordered the XPT_ANN macros to be more
consistent.
I left the padding alone for the enum-like bit flag values, as I think
it makes sense to line those up.
MozReview-Commit-ID: 877aP5eGIFm
--HG--
extra : rebase_source : 6d47ce05b47248c285597454af528ca1ae2cc830
This sounds weird, but either:
1) A method is notxpcom, so it can't be called from script and the XPT
information is unused.
2) A method is not notxpcom, in which case the result type is nsresult.
MozReview-Commit-ID: a7SRJn8PlP
--HG--
extra : rebase_source : 457051a47dd3f1f2f49b5f11ef3e5138f9d814e1
xpt.py only generates constants for a couple of the cases that
XPTConstValue supports. Eliminating the unused cases reduces the size
of this data structure from 64-bit to 32-bit, which reduces the memory
usage of xpti-working set by about 0.02MB.
MozReview-Commit-ID: 1yQkK28fPkR
--HG--
extra : rebase_source : 9d5ce81cfa213cba203ee9c72d2f7dcc8652bd31
num_additional_types is a uint8_t, so its max value is 255. 1 + 255 is
not greater than 256, so the check will pass, but then
num_additional_types += 1 will overflow in the next line.
What I think happened is that bug 1249174 part 6 introduced a bounds
check on an index (which is ok), but then part 8 repurposed this as a
bounds check on the length.
I noticed this because while writing the next patch I ended up with
if (id->num_additional_types > 255)
and then the compiler warned that the check would never fail.
MozReview-Commit-ID: KqiaOyBjj7v
--HG--
extra : rebase_source : 47b20ad2f5e39b05f467cc5b10041070db7fa774
Various strings, like nsISupports, appear many times in XPT data. This
patch adds a cache so we don't write the same string multiple times.
MozReview-Commit-ID: 6buBrXwHqQz
--HG--
extra : rebase_source : 54dea83a9134710c5600828ab68ef3f935f46afd
Andrew Halberstadt
Sylvestre Ledru
Nika Layzell
Andrew McCreight
Andreea Pavel
Dorel Luca