The bulk of this commit was generated with a script, executed at the top
level of a typical source code checkout. The only non-machine-generated
part was modifying MFBT's moz.build to reflect the new naming.
CLOSED TREE makes big refactorings like this a piece of cake.
# The main substitution.
find . -name '*.cpp' -o -name '*.cc' -o -name '*.h' -o -name '*.mm' -o -name '*.idl'| \
xargs perl -p -i -e '
s/nsRefPtr\.h/RefPtr\.h/g; # handle includes
s/nsRefPtr ?</RefPtr</g; # handle declarations and variables
'
# Handle a special friend declaration in gfx/layers/AtomicRefCountedWithFinalize.h.
perl -p -i -e 's/::nsRefPtr;/::RefPtr;/' gfx/layers/AtomicRefCountedWithFinalize.h
# Handle nsRefPtr.h itself, a couple places that define constructors
# from nsRefPtr, and code generators specially. We do this here, rather
# than indiscriminantly s/nsRefPtr/RefPtr/, because that would rename
# things like nsRefPtrHashtable.
perl -p -i -e 's/nsRefPtr/RefPtr/g' \
mfbt/nsRefPtr.h \
xpcom/glue/nsCOMPtr.h \
xpcom/base/OwningNonNull.h \
ipc/ipdl/ipdl/lower.py \
ipc/ipdl/ipdl/builtin.py \
dom/bindings/Codegen.py \
python/lldbutils/lldbutils/utils.py
# In our indiscriminate substitution above, we renamed
# nsRefPtrGetterAddRefs, the class behind getter_AddRefs. Fix that up.
find . -name '*.cpp' -o -name '*.h' -o -name '*.idl' | \
xargs perl -p -i -e 's/nsRefPtrGetterAddRefs/RefPtrGetterAddRefs/g'
if [ -d .git ]; then
git mv mfbt/nsRefPtr.h mfbt/RefPtr.h
else
hg mv mfbt/nsRefPtr.h mfbt/RefPtr.h
fi
--HG--
rename : mfbt/nsRefPtr.h => mfbt/RefPtr.h
The configure option has explicitly thrown an error for more than a year now,
and it happens that the remaining way to still forcefully use it has been
broken for more than 8 months.
DONTBUILD NPOTB
This downloads to a temporary file named uniquely but consistently
based on the URL, and then extracts a build ID using mozversion to use
as a human readable and sortable prefix. This approach can be re-used
by |mach artifact| based Desktop builds.
--HG--
extra : commitid : LxorDuq5D0t
extra : rebase_source : 2f280746f486b79dfe45ad928e4b618e0e12f1a0
The bulk of this commit was generated with a script, executed at the top
level of a typical source code checkout. The only non-machine-generated
part was modifying MFBT's moz.build to reflect the new naming.
# The main substitution.
find . -name '*.cpp' -o -name '*.cc' -o -name '*.h' -o -name '*.mm' -o -name '*.idl'| \
xargs perl -p -i -e '
s/nsRefPtr\.h/RefPtr\.h/g; # handle includes
s/nsRefPtr ?</RefPtr</g; # handle declarations and variables
'
# Handle a special friend declaration in gfx/layers/AtomicRefCountedWithFinalize.h.
perl -p -i -e 's/::nsRefPtr;/::RefPtr;/' gfx/layers/AtomicRefCountedWithFinalize.h
# Handle nsRefPtr.h itself, a couple places that define constructors
# from nsRefPtr, and code generators specially. We do this here, rather
# than indiscriminantly s/nsRefPtr/RefPtr/, because that would rename
# things like nsRefPtrHashtable.
perl -p -i -e 's/nsRefPtr/RefPtr/g' \
mfbt/nsRefPtr.h \
xpcom/glue/nsCOMPtr.h \
xpcom/base/OwningNonNull.h \
ipc/ipdl/ipdl/lower.py \
ipc/ipdl/ipdl/builtin.py \
dom/bindings/Codegen.py \
python/lldbutils/lldbutils/utils.py
# In our indiscriminate substitution above, we renamed
# nsRefPtrGetterAddRefs, the class behind getter_AddRefs. Fix that up.
find . -name '*.cpp' -o -name '*.h' -o -name '*.idl' | \
xargs perl -p -i -e 's/nsRefPtrGetterAddRefs/RefPtrGetterAddRefs/g'
if [ -d .git ]; then
git mv mfbt/nsRefPtr.h mfbt/RefPtr.h
else
hg mv mfbt/nsRefPtr.h mfbt/RefPtr.h
fi
--HG--
rename : mfbt/nsRefPtr.h => mfbt/RefPtr.h
It was added back in
5147d5c69f
for unclear reasons (and the lack of bug number doesn't help), and hasn't been
used, as far as I can see in the gecko-dev history, other than in bug 206029,
which is the only use currently in the tree.
Bug 206029 was working around the Flash player installer modifying Firefox's
prefs file and not dealing with it properly or something depending on the line
endings. 11 years later, all prefs files except channel-prefs.js are in
omni.ja, so obviously, bug 206029 doesn't actually apply anymore.
So, let's simplify it all and get rid of this.
Compressing C++ unit tests is a long pole when writing test archives.
Experimenting with various levels of compression revealed that
compression level 9 was providing minimal space savings for
significantly longer archiving times and greater CPU usage.
Results of our experimentation of `make -sj8 package-tests` on OS X
with various levels of compression are below. Note: these numbers were
accidentally obtained without JS tests being archived. This skews the
results a little but doesn't impact the analysis below.
ARCHIVE SIZE WALL CPU
(L=9)
cppunittest 76,806,629 30.6s
mochitest 61,276,928 9.4s
reftest 31,204,396 11.0s
ALL 228,146,761 31.2s 75.9s
(L=8)
cppunittest 76,851,593 24.1s
mochitest 61,279,322 8.9s
reftest 31,207,867 10.4s
ALL 228,228,096 24.9s 64.7s
(L=7)
cppunittest 77,102,292 14.3s
mochitest 61,305,147 8.2s
reftest 31,260,359 9.4s
ALL 228,717,803 15.0s 49.1s
(L=6)
cppunittest 77,321,408 11.5s
mochitest 61,336,539 8.2s
reftest 31,303,604 9.2s
ALL 229,123,307 12.2s 44.7s
(L=5)
cppunittest 78,226,404 8.2s
mochitest 61,483,804 7.6s
reftest 31,509,349 8.8s
ALL 230,725,600 9.6s 39.7s
(L=4)
cppunittest 79,733,669 6.3s
mochitest 61,825,519 7.6s
reftest 31,924,171 8.4s
ALL 233,669,991 9.0s 36.4s
(L=3)
cppunittest 82,380,731 5.8s
mochitest 62,554,431 7.1s
reftest 32,696,415 8.1s
ALL 239,180,168 8.9s 34.6s
Levels lower than 3 resulted in larger archives with no decreae in
wall time and marginal decrease in CPU time.
As we can see, lowering the compression level reduces archiving time by
>3x while only increasing total archive size by ~2.5 MB or ~1% for
compression level 5.
Total time hits a plateau around levels 4 and 5. After that, file size
increases faster for little decrease in wall time. I suspect that we're
hitting Python limits from having to process thousands of files: there's
only so fast Python can do I/O and make function calls.
I think choosing 4 or 5 for the new compression level are acceptable.
I went with 5 because the wall time savings from 5 to 4 are marginal and
the archive size does start to increase a bit faster at 4. That being
said, 4 does consume 10% less CPU. I could easily just 4 as well. 5 is
more conservative. We can always change to 4 after seeing results in the
wild.
The end result of this change is `make package-tests` is much faster:
Before: 228,146,761 bytes; 31.2s wall; 75.9s CPU
After: 230,725,600 bytes; 11.4s wall; 45.0s CPU
Delta: +2,578,839 bytes; -19.8s wall; -30.9s CPU
When you take the whole series into consideration:
Before: 44.2s wall; 84.6s CPU
After: 11.4s wall; 45.0s CPU
Lowering CPU is impressive considering we switched from the C `zip`
implementation to Python!
Keep in mind we were at ~78s wall before e87b74b3db43 introduced
concurrent archive generation!
And we still haven't eliminated the staging of JS tests, which are
several thousand files and a few dozen MB!
--HG--
extra : commitid : D1fD4NUTw2F
extra : rebase_source : c6de72656cfedc98c0cf1c09eefe1dfb84f3639b
An upcoming commit will introduce a caller that doesn't want the maximum
compression level. This commit introduces arguments to control the
compression level inside written archives.
--HG--
extra : commitid : KkDso3hB2QG
extra : rebase_source : 8fd05aeae5c3555e1169eac6656d584007cd0739
Metrics are nice. Adding this output clearly demonstrates that C++ unit
tests are the long pole by far: they take ~95% of wall execution time
to archive (~30s total). The next longest archive only takes ~11s to
produce. This will be important if we ever want to reduce archive time
further on optimal hardware.
FWIW, disabling compression will produce a C++ unit test archive in
1.0s. Archives with more files take longer, despite the significantly
smaller sizes.
--HG--
extra : commitid : 6E56aUoZUL2
extra : rebase_source : 48cad51d7fbae883861f35e1b5cb96799b452bfb
Won't impact performance much. But fewer make foo makes porting the C++
unit tests (which are the largest remaining tests) to the Python
archiver easier to grok.
This conversion did change behavior slightly. Previously, startup
cache files weren't being packaged if startup cache was disabled. Now,
we always package them since their presence in the test archive should
be harmless. The original change to guard their inclusion in
ee82e0ae5488 was probably unnecessary.
--HG--
extra : commitid : AzU65j0E1q0
extra : rebase_source : 9b8a15dc1a5f3c3d3e453cefb3a99b05f5a77711
This prevents copying of 447 files adding to ~4 MB.
--HG--
extra : commitid : 7zTbiQeMQSQ
extra : rebase_source : b3ac223835ba7289ace45aa7d02c5a050d54cc0d
This saves copying of ~100 files comprising ~1 MB. Not significant. But
it gets us a little closer to no staging.
--HG--
extra : commitid : 6Hjnhv4Yi5R
extra : rebase_source : 291c89682a23cde957b3c68f2efe3b6dc3d3d543
This is slightly more involved than earlier changes because reftests
have a one-off mechanism for finding files. Essentially, the master
reftest manifest is loaded, directories are discovered, and every file
in those directories is packaged.
We add support to our test archive generation tool to read sources from
reftest manifests and tell it where the reftest manifests are.
print-manifest-dirs.py was only being used for staging reftest files.
Since we don't do that any more, the functionality doesn't need to exist
in a standalone file, so it has been moved inline into test_archive.py.
This change avoids copying ~26,000 tests consuming 131 MB during test
packaging. This is a majority of the file count that was remaining in
the stage directory at this point. On my machine (which hasn't typically
seen major wall time wins from not staging files due to its fast SSD),
this change made test packaging ~20% faster, reducing wall time from
~50s to ~40s!
A Try push seemed to indicate drastic results with the series up to this
point. Including the already landed changes to generate test archives
concurrently, test packaging times on OS X builders dropped from ~18:40
to 6:29! Times on Linux x64 remained about the same (~2:46). This is
possibly due to these machines already having SSDs and due to normal
variance in performance of builders and EC2 instances.
--HG--
extra : commitid : 34E8V8lSGg7
extra : rebase_source : 720afcd35f6a2b6cb1217df23ae981408a88cb94
After this, only reftest files themselves are staged. Those will be
addressed in a subsequent commit.
--HG--
extra : commitid : 9jWl9Twcizr
extra : rebase_source : 3e4a319d60b7ee7eddecc597eb250184140b1e71
This avoids copying 5000+ files consuming ~37 MB on my build
configuration.
--HG--
extra : commitid : 6DmsjUYgjXq
extra : rebase_source : 123dd42a7d0b9cc244a3ab7773010dfc5769a4ac
With this change, all test ZIP archives are now generated via Python and
mozpack.
This change does not change I/O or file copy behavior at all. There is
still a lot of room for eliminating extra file copies.
--HG--
extra : commitid : 9mWdtDK6wAb
extra : rebase_source : 0f19c627d64d22bf9d65161d4f7df7c9778dea3c
This doesn't change I/O or copy behavior at all. But it does remove a
one-off make rule.
--HG--
extra : commitid : X0efdFHA0k
extra : rebase_source : c7cb8616461eccd1ff7f8eb3b409bd4944c9e1ec
This is pretty straightforward. This saves ~26 MB of file copies.
--HG--
extra : commitid : ItghoP73zS8
extra : rebase_source : 9656719a6459c1e6fa28165591722fe00d6d9b1d
The web-platform test archive now builds without any staging at all.
This saves ~103 MB of file copies on my machine.
The testing/web-platform/Makefile.in serves no purpose after this
change, so it and all references to it have been removed.
--HG--
extra : commitid : HDHGG3QGVBH
extra : rebase_source : dd7302aad96b46932aa00e4e66918c8077475b10
This is very similar to what we did for xpcshell. Like xpcshell, there
are still some staged files. However, about 73MB of copies are
eliminated with this change. On my machine, overall execution time of
test packaging appears to decrease, although CPU usage is up slightly.
--HG--
extra : commitid : 5dy340X80J9
extra : rebase_source : d37be29367b17e6c1d9c885ab4705932b7a42b39
This commit produces the xpcshell test archive without staging 5000+
xpcshell test files first.
We teach the archiver to ignore .mkdir.done files.
The xpcshell Makefile.in still stages some files. This is less than
ideal. However, it is a small handful of files and shouldn't add too
much overhead.
This appears to not impact overall CPU usage significantly on my
machine, despute using Python instead of `zip`. It does reduce I/O
by ~25MB by avoiding the staging copy.
--HG--
extra : commitid : IwvLaYvAbFt
extra : rebase_source : a690ae4b1adbabd491851a2479fa66d81241601b
Test archive generation currently copies a bunch of files into a staging
area then runs `zip` to produce ZIP files. There are 2 concerns with
this approach:
1) We incur a lot of extra I/O to copy files so everything is
rooted in a single tree so the `zip` invocation and paths are
simple.
2) ZIP files inherit properties from the local filesystem (including
mtime), making ZIP files non-deterministic.
This commit introduces a new mozbuild action for producing test
archives. It does so using the mozpack file finder and JAR writer,
which are used throughout the build to deterministically
produce ZIP/JAR files from files in multiple source directories.
We implement support for producing the mozharness archive. This archive
does not involve files that are staged, so no I/O is saved. In fact,
the switch from `zip` to Python likely makes this slightly slower.
However, we do have deterministic archives now.
Additional archives will be ported over in subsequent commits.
--HG--
extra : commitid : H1BOidPDZST
extra : rebase_source : 120e2bfea921e5fb3a8d97b2dd0227edce452cfd
Previously, we always skipped over files beginning with a ".". This
commit adds an option to include them.
This is needed to support test package generation via Python / mozpack.
--HG--
extra : commitid : 4pmEpukVX0s
extra : rebase_source : 31599230ce344b9be815b3a457cc8a7c6d8e5301
The flags added in toolkit/locales/Makefile.in turn out not to be actually
used, so just remove that.
The remaining uses of XULPPFLAGS are to set debug flags depending on whether
MOZ_DEBUG is set or not. Just set a dedicated variable with the right value
from configure.
When running mach `build-backend` or `config.status`, it is now possible to
pass multiple backends to the --backend/-b option, so that they can share
moz.build reading and object emitting.
The command line syntax is however maybe a little awkward:
mach build-backend -b Backend1 Backend2
but supporting with `-b Backend1 -b Backend2` requires more argument parser
twiddling (action='append' doesn't work out of the box with choices, we'd
need a custom action class)
Currently, we set a flag on each object to know whether it has been consumed
by the backend. This doesn't work nicely when multiple backends try to consume
the same objects.
- Make all backends report the time spent in their own execution
- Change how the data is collected for the reader and emitter such that
each of them is aware of its own data, instead of everything being
tracked by the backend.
This is meant to open the door to multiple backends running from the
same execution of config.status.
This commit exposes test-deps file info as a mach command, and
modifies the test scheme reader to make it filter out unsuitable
contexts when generating TestManifest objects for metadata context.
--HG--
extra : commitid : 7QOoHkfvWOF
This bumps the NDK version to r10e.
Previously, we used brew to install android-sdk and a custom version
of android-ndk. That makes it hard to control the installed versions.
This installs from downloaded archives, which unifies the Mac OS X
approach with the straight-forward Linux approach.
--HG--
extra : commitid : E7hEqsyy8Gw
extra : rebase_source : 9ea27e7d2ae3fbaaa3efbabdd701521981bec877
extra : histedit_source : c07c80c50ac066dc6808e7ccf96f0bc14dc09df2
The 'tools' package depends on 'platform-tools-preview' now. Roll
with it until Google breaks us back again.
The behaviour of the |android| tool has changed; recent versions don't
reveal what packages are installed. That means we can't skip already
installed packages; and we can't really tell if our installation
attempts succeeded. But we have faith!
--HG--
extra : commitid : 341NxbHTJXC
extra : rebase_source : 945e8018effc0b417fc3fedb7220455fabeaedb3
extra : histedit_source : e42fb06e176d5b9e9ebb6553af7045f3a061105f
This gets us a limited version of AAR support: we can consume static
AAR libraries, where here static does not refer to linking, but to
static assets that are fixed at build-backend time and not modified
(or produced) during the build. This lets us pin our dependencies
(and move to Google's versioned Maven repository packages, away from
Google's unversioned ad-hoc packages).
By restricting to static AAR libraries, we avoid having to handle
truly complicated dependency trees, as changing parts of generated AAR
files require delicate rebuilding of the APKs (and internal libraries)
that depend on the AAR files.
It is possible that we will generate AARs in the tree at some time.
Right now, we don't do that, even for GeckoView: the AARs produced are
assembled as artifacts at package time and are intended for external
consumption. We might want this for GeckoView and Fennec at some
time; we should consider using Gradle everywhere at that point.
The patch itself does the simplest possible thing (which has precedent
from Gradle and other build systems): it simply "explodes" the AAR
into the object directory and uses existing mechanisms to refer to the
exploded pieces.
AARs have both required and optional components. Each component is
defined with an expected and required flag. If a component is expected
and not present, or not expected and is present, an error is raised.
If the component is expected and present, autoconf's ifelse() macro is
used to define the relevant AAR_* component variables. If the
component is not expected and not present, no action is taken. A
consuming build backend therefore can guard all AAR_* component
variables with just the top-level AAR variable.
Many AAR files have empty assets/ directories. This patch doesn't
explode empty assets/ directories, protecting against trivial changes
to AAR files that don't impact the build.
There's a lot not to like in this approach, including:
* We need to manually reference internal AAR libs;
* I haven't separated the pinned version numbers out of configure.in.
However, it's closer to what we want than what we have!
--HG--
extra : commitid : 11kUhDAkCn5
extra : rebase_source : 2454c9842ab3296d53ca5fa394a5a962aa382c8d
extra : histedit_source : e2f97502d215016925e93500b8fd93f8b32fba3a
This included adding the slugid 1.0.6 python source code in /python since slugid
is now a dependency of the ./mach taskcluster-graph command, as well us updating
references that used it. Previously the implementation was in-tree.
Previously, we used a PPA on Ubuntu to install Mercurial. The PPA has
proved to be unreliable. Furthermore, we didn't have a mechanism for
installing a modern Mercurial on Debian and other derived distros.
In this commit, we remove the PPA from Ubuntu. We add the ability to
install Mercurial from pip to the Debian bootstrapper. However, since
some people may not want <not apt> installing package-like things,
we add a prompt explaining the situation and giving users a choice.
We recommend installing a modern Mercurial via pip. But we also given
the option to install a likely legacy version via apt. And, for Git
users, we give the option to not install Mercurial at all.
Since the new version of the Ubuntu bootstrapper is empty, it doesn't
need to exist, so it has been removed.
DONTBUILD (NPOTB)
--HG--
extra : commitid : 1n8D8hTG6Fm
extra : rebase_source : 666d9945650c0ce27a32353d98886e2336628a36
extra : amend_source : bb9e7538157fdac0203e2ecc3165185f95717098
Android version codes serve multiple masters. They indicate newer
versions, yes; but they also break ties between versions with
different features and requirements. High order bits effectively
partition the space of versions and are valuable. Since Android
version codes are signed Java integers, we have 31 bits to work with.
Mozilla's traditional build ID is YYYYMMDDhhmmss. This was chopped to
ten characters (YYYYMMDDhh, i.e., hourly build IDs) and then converted
to a decimal. This took many high order bits. We will lose another
high order bit in the 36th month of 2015 -- i.e., as soon as the year
rolls over to 2016. If we waited to lose the next higher order bit,
we'd lose that one in the 46th month of 2017 -- i.e., as soon as the
year rolls over to 2018.
The following patch sacrifices a high order bit to change the version
scheme, winning us roughly 15 years of hourly build IDs before we are
forced to lose another high order bit. So it's clearly to our
advantage to change the scheme sooner rather than later -- we will
sacrifice 1 bit for 15 years of build IDs, rather than keeping the
current scheme and sacrificing (say) 2 bits for 3 years of build IDs.
The resulting scheme produces build IDs that look like (in binary):
0111 1000 0010 tttt tttt tttt tttt txpg
The meaning of these build IDs is documented in the Python source code
that generates them.
--HG--
extra : commitid : 7BEkkGHsYVL
extra : rebase_source : bbc0ead8e7a383c320e838b023b02b1fb0d94ff3
extra : amend_source : dab8c737e3274694aad40123e77884a3239908de
extra : histedit_source : 530bedde3695d534805465cdf8bcb9cd23f7b031
The patch removes 455 occurrences of FAIL_ON_WARNINGS from moz.build files, and
adds 78 instances of ALLOW_COMPILER_WARNINGS. About half of those 78 are in
code we control and which should be removable with a little effort.
--HG--
extra : rebase_source : 82e3387abfbd5f1471e953961d301d3d97ed2973
This moves a little bit more of mobile/android/base/Makefile.in into
moz.build, and gets closer to moving that aapt invocation into
java-build.mk.
There are no other extra package consumers in the tree. (There should
be a new one shortly: b2gdroid.)
--HG--
extra : commitid : AaYqXYReOSX
extra : rebase_source : d41368ff0bd0736221fdc04ed8299b70c2488c8b
extra : histedit_source : 845efd5ba9f99f4e186c3a5c66affe69eac7fec7
This paves the way for defining additional Android packages in
moz.build, which is a step toward moving the special
mobile/android/base/Makefile.in aapt invocations into the generic
java-build.mk framework.
The new variables are both passthru variables for now: in the future,
we'll roll them into some aggregate Android APK definition.
It's worth noting that references to the variables in Makefile.in
files are only defined after including rules.mk (and thereby
backend.mk). This only required a few changes in the tree but it
confused me for some time.
--HG--
extra : commitid : G5mEvm8Ng4F
extra : rebase_source : 7ba05f2e53554549ffb5cefe270925e3e2025b6a
extra : histedit_source : eacd22f4b7edddab67147c413fea45a3ba292c0c
Over a 19 minute build, |mach build| was spending ~45s drawing the TIER footer. This brings it down to ~30s.
--HG--
extra : commitid : 2sLWbNL853A
extra : rebase_source : ab8452dfa6ac39bc40a8f22b542e155595f18881
This change adds a 'what' parameter to |mach clobber| which facilitates clobbering other things as well.
E.g, |mach clobber python| now clobbers all .pyc and .pyo files in the tree. Multiple clobber targets can
be specified, e.g |mach clobber objdir python|. By default |mach clobber| without arguments will only
remove the currently active objdir, the same as before.
--HG--
extra : commitid : 4mJqdnWPRbH
extra : rebase_source : cf56de2d51c57747047936f7e742bafbf3f91c9f
We have had singular ANDROID_ASSETS_DIR in Makefile.in for a while.
Fennec itself does not use the existing Makefile.in Android code, for
complicated historical reasons.
This makes the existing variable moz.build-only; generalizes the
existing variable to an ordered list; and adds the equivalent use of
the new list to the Fennec build, with a simple example asset.
This patch also updates the packager to include assets packed into the
gecko.ap_. Without the packager change, the assets/ directory in the
ap_ gets left out of the final apk. This whole approach is totally
non-standard but is more or less required to support our single-locale
repack scheme.
--HG--
extra : commitid : 4EAh1UNGNWT
extra : rebase_source : 5e5b4c4a120c3b4cc776c9f9380ddd2f9b63587e
extra : source : 0ddce3eb833e6d6180a19928a9b45d5d12f1d7fa
This patch does a few things. First, it adds an AbsolutePath data
type, sibling to SourcePath and ObjDirPath. (Existing Path consumers
that accept an open set of Path subtypes, and that only use full_path,
should function fine with the new AbsolutePath subtype.)
Second, it moves ANDROID_RES_DIRS to a moz.build list of Paths
(ordered). We test, but don't use in tree, the new AbsolutePath.
--HG--
extra : commitid : DMLy1ogTJ0Y
extra : rebase_source : cb9ac47e8bf7c893a0284adc7a42eccb78ccae3d
Mach has special handling for remainder arguments which previously
only worked with the decoration form. Extend this to also work when
a command passes in an explict parser
--HG--
extra : commitid : ICnYk764Xbc
extra : rebase_source : 1518edcb7bef375cbc2931b8da54b015045c8ce1
It is useful to be able, during mozconfig execution, to do tests depending
on what was previously added with mk_add_options. Specifically, there is a
need to do this for MOZ_PGO because developers pushing to try may add it to
mozconfig.common.override.
While, ideally, it would be nice if we just defined the variable itself in
the mozconfig execution environment, that is a tedious task, having to jump
through hoops with eval, and handle all cases of variable assigment properly.
The hacky alternative is to just treat MOZ_PGO specially, but meh.
So instead, we set a ${var}_IS_SET variable to 1, indicating that a
mk_add_options defined ${var} to some value.
Bug 1153566 changed the regexp used in that method in such a way that there
has been a big hit in the time spent executing the make backend. On my machine,
with the current code, mach build-backend indicates:
Backend executed in 5.01s
With the change from bug 1153566 reverted:
Backend executed in 2.97s
That's a significant regression for a 4-character change.
But we can actually avoid using regexp in most cases, which can make things faster
than they were. With this change, we get down to:
Backend executed in 2.28s
For reference, making the _check_blacklisted_variables method do nothing at all
ends with:
Backend executed in 2.20s
DONTBUILD NPOTB on a CLOSED TREE
--HG--
extra : commitid : 2ft26PqDgyT
extra : rebase_source : ca2eb4fe7c04e97fb78cd2175bd8549e13c13d7e
extra : amend_source : 10a2fa5931d151af448269239bdf2dc1980d3fae
This just allows a little versatility for consumers such as b2gdroid,
which are Fennec-like but don't have MOZ_APP_NAME=fennec.
I elected to pass appname as a parameter rather than modify the
existing distdir because I expect to want to differentiate, in some
way, the output AAR files based on the underlying name. That is, in
future we might generate geckoview-fennec-VERSION.aar and
geckolibs-b2gdroid-VERSION.aar, or stuff the name into the Ivy version
information, or...
This also fixes a typo in one of the JarFinder instantiations.
--HG--
extra : commitid : CnJKouGgkh1
extra : rebase_source : 5767e66ea53e14dd6468adec741773a02a6e2d3a
CalledProcessError.output and subprocess.check_output's return value
are str types. This file uses unicode_literals. If we do something
like `if 'foo' in e.output`, there will be a mix of str and unicode
types and Python will do implicit conversion. If the strings aren't
ASCII, we'll likely encounter a UnicodeDecodeError.
Use b'' literals around all strings to prevent this coercion from
occurring.
--HG--
extra : commitid : 89l5oW9T8ks
extra : rebase_source : f402353cd4a759c882475518f1360687af1936dc
extra : amend_source : ec31a39620eae2c90fb832f314cd161905444a34
This will give us the ability to execute custom code at command dispatch
time. We can use this for global tests before dispatch.
--HG--
extra : commitid : 5vfmSqOis4W
extra : rebase_source : c285b16007c4b604b164d079a275198c8760e480
I considered three ways to do this:
* one, as a Python script executed with $(shell);
* two, as a Python script that writes an include file for the preprocessor;
* three, as a function exposed to the moz.build sandbox.
I rejected two because it's both tied to the preprocessor, and awkward
to make handle the dependency on the buildid (in a file) and
additional build defines (in config.status).
I rejected three because I know of no precedent for this approach, and
it hides the dependency on the buildid.
One doesn't handle failures in the script gracefully, but neither did
the existing approach. This patch is at least testable.
--HG--
extra : commitid : 8dfw1ri7qjr
extra : rebase_source : da0e5ec705e0ac4c795bd2d7892f73857a1699ac
Without this, we throw UnrecognizedArgumentError when running commands
such as `./mach mochitest-plain test`, which causes an error message
such as the below to be emitted:
It looks like you passed an unrecognized argument into mach.
The mochitest-plain command does not accept the arguments: test
This patch will fix the above command to instead print the corresponding
deprecation message.
DONTBUILD NPOTB
This was just an oversight during the initial landing, leading to two
copies of artifact libraries being appended to the same destination
file.
--HG--
extra : commitid : JvvZDrjUOQZ
extra : rebase_source : f48c1a9d4b506b6ed931043aeeaca437418ea9c3
extra : histedit_source : 3dbfd049039c0adc595c1abb0df3dca3af9db207
This implements a new "scan" mode for DMD that records the address
and contents of every live unsampled block in the DMD log. This
enables the low-level analysis of references from one block to
another, which can help leak investigations.
bf34d16b6ab2 added absolute_import to this file. When changed, "import
mozinfo" stopped picking up mozbuild.mozinfo and started importing
mozinfo instead.
Use relative imports to force mozbuild.mozinfo to be picked up.
--HG--
extra : commitid : 4GC5lJWrBFq
extra : rebase_source : c91674b16f29dadf8dcda460787ea94654f2864a
This is a regression from bug 1176620 that results in all Firefox mach commands showing up in the help for B2G, even though they don't work there.
--HG--
extra : rebase_source : e779b866a82c2df1dbf913d24b93a0dc9838ff9f
This was the only import of glob from all mach_commands.py files. Kill
it.
With this commit, there are no modules imported by a single
mach_commands.py outside of testing/web-platform/mach_commands.py.
--HG--
extra : commitid : 4CJqlwDqOVg
extra : rebase_source : 9dbbd69291d64b894a399523864562107c10872e
This import brought in significant parts of the mozbuild package. Moving
it to a deferred import reduces the total number of Python modules
imported during mach dispatch by 43.
--HG--
extra : commitid : GdXsF7agvCT
extra : rebase_source : 586f1960c8e7eb400f61467045b064167784f68b
This removes ambiguity as to which modules are being imported, making
import slightly faster as Python doesn't need to test so many
directories for file presence.
All files should already be using absolute imports because mach command
modules aren't imported to the package they belong to: they instead
belong to the "mach" package. So relative imports shouldn't have been
used.
--HG--
extra : commitid : 6tFME1KKfTD
extra : rebase_source : 78728f82f5487281620e00c2a8004cd5e1968087
DONTBUILD ON A CLOSED TREE: Android-only and the build changes are cosmetic.
Very much a first cut, but I'd like to get some Fennec early adopters testing.
This adds:
* |mach artifact install| to fetch and install Fennec binaries;
* |mach artifact last| to print details about what was last installed;
* |mach artifact {print,clear}-caches|, for debugging.
This code is exposed as a new mozbuild.artifacts Python but it's not
particularly general. My intention was to get things out of the mach command
more than produce a general artifact fetching API. We can leave that bike
shed to Bug 1124378.
I've been testing this with --disable-compile-environment and it works well
locally, although there's no reason a knowledgeable developer couldn't use
this in conjunction with a fully-built tree. (I don't know when such a
situation would arise, but I know of no technical impediment.)
--HG--
extra : commitid : 1T28aVfArqF
extra : rebase_source : b8c11244de8be0a14d605853f30cd47312d0a4ba
extra : histedit_source : 78a224501cd3cf0f86707c9c9549b61b4b248ba7
This type is now redundant with our new rich type for capturing all mach
command metadata. Eliminate it and using _MachCommand instead.
--HG--
extra : commitid : 7pXhf6V7F8m
extra : rebase_source : 343615096f23d3acf23f7487c7b8c7137c85337e
Simplify construction of mach's MethodHandler instances by by passing in
our new rich type that holds all command metadata.
While we are here, kill the docstring argument, as it can be computed
easily inside MethodHandler.__init__.
--HG--
extra : commitid : I5PRlYDVXVq
extra : rebase_source : ebf07116357036ddfec06fd32fb161fefff628e6
Up to this point, mach command metadata has been stored in tuples.
Initially, things weren't so bad. But they have evolved into tuples with
many elements. Adding new attributes is cumbersome. Let's restructure
the code to capture metadata in a dedicated class.
Before, there existed a separate attribute on the @Command or
@SubCommand decorated method for each mach decorator: @Command,
@CommandArgument, @CommandArgumentGroup. With the magic of __ior__,
we can now capture all metadata on a single type. This simplies
processing, as we now only look at a single attribute on methods:
_mach_command.
Before, we used separate attributes to distinguish between mach commands
and mach sub-commands. Now that we have a type that can hold all data,
we combine things into the _mach_command attribute and look for the
presence of the "subcommand" attribute on this type to identify
sub-commands.
--HG--
extra : commitid : EtBwUmS5TV2
extra : rebase_source : 4ef5f95a532693672c6c4b33fa1c22adb76d105e
We want this logic to live next to where metadata types are defined so
downstream consumers of Files-based metadata don't have to reinvent the
wheel.
The work in this commit will be used to enable auto filing bugs during
pushes to MozReview.
--HG--
extra : commitid : F55RzhDCVmR
extra : rebase_source : 1fe1a03f8d592c60d2ed8d760fd62c7bc60421a5
This file hasn't been updated in ages and the current configuration
doesn't produce working packages. Change that.
--HG--
extra : commitid : AfiwohniLPq
extra : rebase_source : 70786c7bd9e37c9acc9b38f54c956fec424cfcf5
extra : amend_source : e4e3752dc97b934456b973736ef55633366c8ee6
This should have been done in 4c757e339f81.
DONTBUILD (NPOTB)
--HG--
extra : commitid : 41TCyM9iqVQ
extra : rebase_source : 8defc6b23d507e1580db77869ff2c2e99ee5db5a
extra : amend_source : 5f330ea7c2775675fcc8051e5c46d2439c4e5823
CentOS has apparently changed its disto name. Detect the new version.
DONTBUILD (NPOTB)
--HG--
extra : commitid : JrYp4WJ5Fs3
extra : amend_source : c2ac16f7006c2e44450de54edb732fa722f150d0
The hglib Mercurial finder was nice. But it is somewhat slow, as it
involves a separate Mercurial process.
This commit introduces a native Mercurial finder that speaks directly
to a Mercurial repository instance. It is significantly faster.
The new code is isolated to its own file because it imports Mercurial
code, which is GPL.
--HG--
extra : commitid : 8CzDFt3lQmx
extra : rebase_source : 1540ad20d795086926b316612062e2a1f10c4958
This commit adds support for specifying a Mercurial revision with `mach
file-info`. Aside from being a potentially useful feature, it proves
that MercurialRevisionFinder works with BuildReader.
--HG--
extra : commitid : 7143XT9ENqb
extra : rebase_source : d8c0c98d536d07db76653b648fd4b7d74d8f43ae
The moz.build reader uses absolute paths when referencing moz.build
files. *Finder classes expect paths to be relative to a base. When we
switched the reader to use *Finder instances for I/O, we simply provided
a default FileFinder based at the filesystem root. This was quick and
easy. Things worked.
Unfortunately, that solution isn't ideal because not all *Finder
instances can accept absolute paths like that. The
MercurialRevisionFinder is one of them.
Changing the moz.build reader to talk in terms of relative paths is a
lot of work. While this would be ideal, it is significantly easier to
defer the problem until later and hack up MercurialRevisionFinder to
accept absolute paths. This commit does exactly that.
Bug 1171069 has been filed to track converting BuildReader to relative
paths.
--HG--
extra : commitid : 2JmzOBldBLy
extra : rebase_source : a8af6ce88dd9e2b98f131c92b45c3ece852e13d2
Now that moz.build files use finders for I/O, we can start reading
moz.build data from other sources.
Version control is essentially a filesystem. We implement a finder
that speaks to Mercurial to obtain file data. It is able to obtain
file data from a specific revision in the repository.
We use the hglib package (which uses the Mercurial command server) for
speaking with Mercurial. This adds overhead compared to consuming the
raw Mercurial APIs. However, it also avoids GPL side-effects of
importing Mercurial's Python modules.
Testing shows that performance is good but not great. A follow-up
commit will introduce a GPL licensed Mercurial finder. For now, get
the base functionality in place.
--HG--
extra : commitid : BkwaQOW9MiR
extra : rebase_source : 915d6015317ccc79c228a76eed861d9f43e2fd17
Sometimes moz.build data may not come from the local filesystem. To
support defining moz.build data in other backing stores, we switch
moz.build reading to use mozpack's *Finder classes for I/O.
The use of a FileFinder bound to / is sub-optimal. We should ideally
be creating a finder bound to topsrcdir. However, doing this would
require refactoring a lot of path handling in the reader, as that code
makes many assumptions that paths are absolute. This would be excellent
follow-up work.
While I was here, I cleaned up some linter warnings for unused imports.
On my machine, this commit results in a slight slowdown of moz.build
reading. Before, `mach build-backend` was consistently reporting 1.99s
for reading 2,572 moz.build files. After, it is consistently reporting
2.07s, an increase of 4%. This is likely due to a) function call
overhead b) the cost of instantiating a few thousand File instances
c) FileFinder performing an os.path.exists() before returning File
instances. I believe the regression is tolerable.
--HG--
extra : commitid : 1WDcrSU3SQD
extra : rebase_source : a23aa5a4063c6f7080ee00b6f0fe0ee532da3ce7
Passing raw file handles around is a bit dangerous because it could lead
to leaking file descriptors. Add a read() method that handles the simple
case of obtaining the full contents of a File instance.
This should ideally be a method on BaseFile. But this would require
extra work and isn't needed. So we've deferred it until bug 1170329.
--HG--
extra : commitid : 82qw76XmpjC
extra : rebase_source : 422b16c5a3b1577f080097925aeaeb560aa3e798
Today, the *Finder classes are optimized for doing matching and
returning multiple results. However, sometimes only looking for a
single file is wanted.
This commit implements the "get" method on all the *Finder classes.
It returns a BaseFile or None.
FileFinder._find_file has been refactored into FileFinder.get
to avoid code duplication.
--HG--
extra : commitid : K9It2ZJ3Rbo
extra : rebase_source : a56f8f70aa1902d26373a7196eae2847cce653d3
The current mode of execution of templates doesn't allow them to more advanced
control flow, like returning early, returning or yielding values, because that
mode of execution is equivalent to running the code at the top level of a .py
file.
Making the templates executed through a function call, although trickier,
allows those control flows, which will be useful for template as context
managers.
inspect.getsourcelines() and inspect.getfile() involve I/O out of our control.
Our use of those functions, however, doesn't require all their smarts. In fact,
we only use them on function objects, for which we can just do the work
ourselves without involving inspect functions that trigger I/O.
`mach help <command>` currently only displays a brief description of the
command along with its arguments. Sometimes more detailed help text is
needed.
With this commit, the docstrings of mach command handlers will appear in
the output of `mach help <command>` if they are defined.
I've implemented basic docstrings for the three flavors of mach commands
(normal command, main subcommand, subcommand) to demonstate things work.
My hope is others will start to fill in docstrings once this feature
lands so the output for `mach help` can serve as a better learning guide
for new contributors.
--HG--
extra : commitid : Hx6ZkHDxbCK
extra : rebase_source : 01ced5a044442e370a45cd3fb245ac6283316925
extra : amend_source : fceb771e0e1ffa4e6f3f1b7c22eae6e25cf82034
The old way of writing scripts for generated files would invoke the script thusly:
python script.py arg1...
Invoking the script this way means that the script's directory is
automatically added to sys.path, and importing modules from that
directory is easy. Let's make it equally easy in the new world for
GENERATED_FILES, too.
Document the role of l10n.ini, filter.py, file paths and checks.
Also add a glossary for the jargon used in the doc.
--HG--
extra : rebase_source : ec71f9b7123ba76370d34d2349b2f078f14dd1de
Add generic support for different forms of paths in moz.build:
'/topsrcdir/relative/paths'
'srcdir/relative/paths'
'!/topobjdir/relative/paths'
'!objdir/relative/paths'
This drops the use of UserString for performance reasons, which required
going around with a meta class.
This code has been commented since it landed, and enabling it requires
adding the proper conditionals around all non_unified_sources in gyp
configurations, which is a daunting task. OTOH, the commented code is
already outdated (it would need updates to work) and the related code
that is not commented gets in the way of upcoming changes, so remove
it.
This helps upcoming changes, and relieves backends from path resolution.
This has the side effect of chaning the order of some unified sources,
which consequently breaks nsTextFormatter because it declares snprintf
methods after nsStringAPI #defines it.
Andrew Halberstadt
Nathan Froyd
Tooru Fujisawa
Mike Hommey
Nick Alexander
Wes Kocher
Carsten "Tomcat" Book
Sebastian Kaspari
Gregory Szorc
Michael Ratcliffe
Ehsan Akhgari
Chris Manchester
J. Ryan Stinnett
Xidorn Quan
Ted Mielczarek
Peter Moore
Geoff Brown
Nicholas Nethercote
Birunthan Mohanathas
Philipp Kewisch
Joel Maher
Niels De Graef
Aaron Raimist
David Keeler
Ryan VanderMeulen
Eric Rahm
Kyle Huey
James Graham
Allison Naaktgeboren
Dustin J. Mitchell
Vladimir Vukicevic
Mitchell Field
Ben Hearsum
Supradeep T R
Jonathan Griffin
Peter Elmers
Julien Pagès
Nicholas Hurley
Edmund Wong
Andrew McCreight
Timur Timirkhanov
Benjamin Smedberg
Gijs Kruitbosch
Michael Comella
Axel Hecht
Mike Hommey
Kartikaya Gupta
Brian O'Keefe
Joe Steele