This adds support for an SDK_FILES variable in moz.build, which creates
a FinalTargetFiles object to install files into dist/sdk/
MozReview-Commit-ID: 97a5NdbZmmD
--HG--
extra : rebase_source : ad8d521ec56fe4610437c8d2d503c545894b40c2
The changes made to PATH by activating the in-tree virtualenv cause some git-cinnabar commands
in subprocesses to fail, because git cinnabar is expecting mercurial to be installed, while it
is not available from the in-tree virtualenv. To work around this, the changes to PATH made by
virtualenv are undone, allowing git-cinnabar to find system python, which is likely to have
mercurial installed.
MozReview-Commit-ID: 47ceHvMmuyf
--HG--
extra : rebase_source : 199d3b01fc49b2760ee14a08d4400a1addf83eac
On a CLOSED TREE because DONTBUILD NPOTB
MozReview-Commit-ID: 56vyz2CRJsU
--HG--
extra : amend_source : 4ec60bc95019147225479c32b6982dc33c649cc4
extra : histedit_source : c3dc78da75a8f5b3985024a7d73ac92ab80628c2
This moves all the reading mozconfig, finding autoconf, refreshing the
old configure, and running the old configure into sandboxed
moz.configure. This effectively bootstraps the sandboxed python configure.
Generally speaking, the configuration needs forward-slashes in paths.
We might as well make it hard(er) to set configuration items with
backslash separators on Windows by exposing mozpath.* functions in place
of os.path functions. The downside is that functions explicitly
importing os will still get the real os.path functions.
The upcoming move of the configure.py initialization to sandboxed
moz.configure changes the path separators for topsrcdir and topobjdir
from native to always use forward-slashes, which confuses the hell out
of the test_base.py test. Settle the issue by declaring that
MozbuildObject will always use forward-slashed paths for topsrcdir
and topobjdir.
The nice side effect is that now we can have actual dicts for defines
and substs from the start, which simplifies so things, although it
requires adjustments to some unit tests.
Creating a .metadata_never_index file in a directory apparently
disables Spotlight indexing.
MozReview-Commit-ID: Di4DGZK6VOL
--HG--
extra : rebase_source : feffa0785174b11cf88a2b7ffdbd9aa5eeb98ac7
extra : amend_source : 838aca018ab91720b6b29b64b846a3ef107b6c88
Using ccache can have a big impact on compile times but we don't currently
capture ccache statistics.
MozReview-Commit-ID: CdrScyAh64I
--HG--
extra : rebase_source : 70dca7ae153ae1c7ce3a241820ab7d18cb6aaa0d
We need to consider CONFIGURE_SUBST_FILES as generated files when
processing FinalTargetFiles and related variables. Additionally, we have
to make sure that we actually recurse into such directories during the
export phase.
MozReview-Commit-ID: 6ZwHMzjoT6t
--HG--
extra : rebase_source : e39e26ad8b84bcaa757c56048c9c80f5be270d9c
The Windows content indexing service has been known to scan the objdir.
This can add significant system overhead and slow down builds or
subsequent operations. The objdir is meant to be a short-lived black box
and there really isn't a major benefit to indexing it.
There is a file attribute on Windows that disables content indexing.
This commit adds a utility function for creating a directory and
optionally disabling indexing on it. We call it at the top of
`mach build` to ensure the objdir as content indexing disabled.
MozReview-Commit-ID: 68gxCxbkVAN
--HG--
extra : rebase_source : 8e95d9b2923dccadbe54288bea25883937e2f004
We don't process IPDL and WebIDL files during artifact builds. The
backend shouldn't need to care about them.
Before: 2001 total backend files; 2001 created; 0 updated; 0 unchanged; 0 deleted
After: 1945 total backend files; 1945 created; 0 updated; 0 unchanged; 0 deleted
After this change, we no longer write any .cpp files to the objdir
during config.status for artifact builds.
As part of this, we stopped generated webidlsrcs.mk and the build broke
because of an unguarded use in a Makefile.
After this change, only 102/3366 files in the objdir after
`mach configure` are not a) in _virtualenv b) named backend.mk
c) named Makefile (~950 each of backend.mk and Makefile).
MozReview-Commit-ID: 11AIn1i4x4f
--HG--
extra : rebase_source : ac90bb9f6be8b7b986aa0a61b3ccc203a18346a6
I can't think of a good reason why unified C++ files need to exist
during artifact builds. Let's not write them in this build config.
Before: 2517 total backend files; 2517 created; 0 updated; 0 unchanged; 0 deleted
After: 2001 total backend files; 2001 created; 0 updated; 0 unchanged; 0 deleted
No measureable change in performance on Linux. But on Windows where file
creation is in the ~1ms range, this will surely result in a speedup of
several hundred milliseconds.
We are still generating some .cpp files during artifact builds. This will be
addressed in subsequent commits.
MozReview-Commit-ID: Lqx36YE8qZZ
--HG--
extra : rebase_source : 8af5a9915e78af7aee2aa335552689fe33975400
Future commits will add a number of consumers. This will cut down on
boilerplate.
MozReview-Commit-ID: 8h4VWBXXd8O
--HG--
extra : rebase_source : 3376f015f1766eedcef60e9393b6d68474ffb634
As previous measurements have shown, creating/appending files
on Windows/NTFS is slow because the CloseHandle() Win32 API takes
1-3ms to complete. This is apparently due to a fundamental issue
with NTFS extents. A way to work around this slowness is to use
multiple threads for I/O so file closing doesn't block execution
as much.
This commit updates the file copier to use a thread pool of 4
threads when processing file copies. Additional threads appear
to have diminishing returns.
On my i7-6700K, this reduces the time for processing the tests install
manifest (24,572 files) on Windows from ~22.0s to ~12.5s in the best
case.
Using the thread pool globally resulted in a performance regression
on Linux. Given the performance sensitivity of manifest copying,
I thought it best to implement a slightly redundant non-Windows
branch to preserve performance. For the record, that same machine
running Linux is capable of processing nearly the same install
manifest (24,616 files) in ~2.2s in the best case.
MozReview-Commit-ID: B9LbKaOoO1u
--HG--
extra : rebase_source : e9fee3861a70e1da9d18448f8435f4bd3e28c647
This adds a simple schema for build telemetry data. We can make it more
restrictive once we have a better feeling for what kind of data we want
to submit.
This also moves more common data about the system to the telemetry handler.
We leave psutil derivied information in the resource usage data as not every
system will have psutil installed.
MozReview-Commit-ID: CFRq1Ow6AOf
--HG--
extra : rebase_source : 3022d8f5d20e3d4f9dc871cf2217a6dad2f22e05
self._pushhead_cache no longer exists. But self._tree_cache does!
This was causing AttributeError when running `mach artifact clear-cache`
and other misc `mach artifact` sub-commands.
MozReview-Commit-ID: CP8NL6eCfhD
--HG--
extra : rebase_source : 0afd11722e304c8e0ecd9a305023d43dff79dddd
extra : amend_source : ad3df6d780e7b968573588e9a1029f1a1c9d18b0
Maja Frydrychowicz
Mike Shal
Chris Manchester
Wes Kocher
Nick Alexander
Sambuddha Basu
Mike Hommey
Myk Melez
Nicholas Nethercote
Gregory Szorc
Dan Minor
Quentin Headen
Jan Beich