When investigating why a particular build got a particularly high number
of cache misses, it is useful to know exactly *what* had cache misses,
and you can't get the missing information after the fact.
We used to have some stats stored in a log file uploaded as artifact
with the old python sccache per bug 1005495 (and before that inline in
the build log), and we actually still have remains of that in the form
of a EXTRA_UPLOAD_FILES in build/mozconfig.cache.
Let's reuse that, and get some useful logs from sccache. Ideally sccache
would give better logs, but those are sufficient for now. We'll figure
things out eventually. https://github.com/mozilla/sccache/issues/151
--HG--
extra : rebase_source : a2e7093697a5b720ab28f155560211d6c0a4347e
Thunderbird is still using MOZ_PKG_PRETTYNAMES which has spaces in the package
name, which gets mangled by abspath.
MozReview-Commit-ID: 23ngx6gIESI
--HG--
extra : rebase_source : 9038f295e582bec5872898a5e8f052bfacf8f85a
This commit makes sccache dump JSON stats at the end of the build, and then
reads them in `BuildScript.generate_build_stats` and adds them to the
build_metrics we submit to Perfherder. The stats dumping is done in
Makefile.in where we currently dump verbose sccache stats because sccache
doesn't persist stats to disk right now and it will also shut down its server
process after 5 minutes, so when the post-build automation steps take more
than 5 minutes the server shuts down and the stats are lost.
Currently it's collecting:
* Cache hit rate
* Cache write errors
* Non-cacheable requests (compiler invocations that sccache can't cache)
We can always grow this list later.
MozReview-Commit-ID: J9CwU7XB05I
--HG--
extra : rebase_source : 084b09c3b0621330ac331a99b1bca9a15cf833b7
This commit moves symbol dumping to the compile tier, to be run via "syms"
targets. Tracking files are used for the sake of incremental builds, because
dump_syms may genearate multiple outputs whose paths are not known ahead of
time.
Minimal changes to symbolstore.py are made here. More extensive
simplifications will be made in a future commit on the basis of symbolstore.py
handling one file at a time.
MozReview-Commit-ID: 3mOP8A6Y7iM
--HG--
extra : rebase_source : bfe97afcdfc05b9e79f01577701c83e8b00eb4e9
This removes the UNIFY_DIST and UNIFIED_BUILD variables, as well as the
--unify flag from the packager and UnifiedBuildFinder from mozpack. As a
result the STAGEPATH variable is never defined anymore, so its uses can
be removed as well.
test_unify.py is currently the only mozbuild/mozpack test that fails
without running configure first, and there isn't much point in fixing
tests for things that we don't actually use anymore.
MozReview-Commit-ID: F5q1FPW3Did
--HG--
extra : rebase_source : cadbd237f51c23ea1983135294521d628d16f0df
This replaces the 'run-tests-deps' make target with a python function that will directly
read moz.build files, emit them with TestManifestEmitter, then consume them with
TestManifestBackend. Because the TestResolver is the only place that actually reads the
test metadata files, we can remove this logic from the CommonBackend as well.
MozReview-Commit-ID: DXgMoeH5dKf
MozReview-Commit-ID: HstZ57qkqf2
--HG--
extra : rebase_source : f377fa6863ef66d3adb86ed64f844e346686862f
These automation steps are no longer relevant, since the last use of
MOZ_AUTOMATION_PRETTY was removed along with non-unified builds in bug
1121000.
MozReview-Commit-ID: 11VDu8pIs1p
--HG--
extra : rebase_source : a82b52400b73224e06e124dd10d6646d4b21a7fc
This updates the client artifact code to locate test support files in the
common test archive and populate the objdir with these files appropriately.
MozReview-Commit-ID: GuXjwUtsl
--HG--
extra : rebase_source : 3560efee22533f60be1e7394fa0841005991cca6
This updates the client artifact code to locate test support files in the
common test archive and populate the objdir with these files appropriately.
MozReview-Commit-ID: GuXjwUtsl
--HG--
extra : rebase_source : 3560efee22533f60be1e7394fa0841005991cca6
In bug 1320194 we want to generate a TestManifest backend without first
invoking configure. However, we still need a way to update the backend,
which is normally handled by the top-level Makefile. We can split this
logic out into a separate file, rebuild-backend.mk, which can be invoked
directly as long as the appropriate environment variables are set
(BUILD_BACKEND_FILES, BACKEND_GENERATION_SCRIPT, and PYTHON).
--HG--
extra : rebase_source : b6a486db3e3c97b406b11074fda052bd63fe1b8f
This patch does a few things:
1) Change all the in-tree tooltool manifests to contain sccache2 instead of the existing Python sccache
2) Change mozconfig.cache to point at sccache.
3) Lightly tweak the --with-cccache configure option to support sccache, and detect whether we're using ccache or sccache and set an option appropriately.
4) Add a MOZ_SCCACHE_VERBOSE_STATS option, and add a target in the top-level Makefile to make sccache spit out its stats at the end of the build. This is useful to see the cache hits/errors until we get something better.
5) Add MOZ_USING_SCCACHE to the build telemetry. Not that I think it will be immediately useful, but for future use.
MozReview-Commit-ID: 9lrdLwNj5Bm
--HG--
extra : rebase_source : d323457df10d0ee0ac5811940e518d9422a7e070
Each archive takes 10+ seconds to generate. Split the archiving
part into separate make targets so this completes faster.
I'm not super thrilled about adding yet more make targets to the
root Makefile.in. We can refactor this later. For now, let's
take the quick win.
MozReview-Commit-ID: 2Oeq1rX6yLw
--HG--
extra : rebase_source : 55282a1dacf65adcaf559c178c61ae42591b29ea
This also fixes the issue of processing the artifacts twice in some
situations (bug 1275673). Note that the artifact download no longer
happens when a specific target is passed to 'mach build'.
MozReview-Commit-ID: Ktys6u3r1kG
On my i7-6700K time and size for the full zip archive for a Windows
debug build is as follows:
Time Size
Before: 25.4s 278,708,278
After: 13.0s 280,623,521
Anything lower than level 5 and the archive size starts increasing by
several megabytes. Level 5 is definitely the sweet spot.
MozReview-Commit-ID: 9lW2W1htq93
--HG--
extra : rebase_source : 26458386e4032dfd85eddd937bf6e6bff057d99b
This moves test installation for test files out of the monolithic install
manifest for $objdir/_tests, and determines the test and support files
to install based on the object derived from all-tests.json. Additionally,
the files resulting from TEST_HARNESS_FILES are installed, as some tests
will depend on them.
As a result, the time to install tests when invoking the test runner will
scale with the number of tests requested to run rather than the entire set
of tests in the tree, resulting in significantly less overhead.
MozReview-Commit-ID: LeIrUVh1yD4
The behavior is not entirely idempotent (most notably for
buildconfig.html), but this can be improved later if necessary.
It is idempotent where it matters.
This allows to get rid of config/makefiles/rcs.mk and its uses.
FasterMake needs some RecursiveMake install manifests to have been
processed before doing its work, so we can actually end up processing
them twice because of the going back and forth from FasterMake in the
hybrid build system.
Set the dependency at the RecursiveMake level when doing an hybrid
build.
--HG--
extra : commitid : 7nD60DTsoHz
extra : rebase_source : 61b5803732b0ecdff421e4e15a2086d4eae7a937
extra : amend_source : c0ae62708b2019888ea320c3793d4ea3f6d6d460
For some reason, recent changes with the install-dist/% rules broke the
dummy wrapper used by `mach build faster`, leading to `Nothing to be done
for install-dist_idl.`, subsequently failing to build xpt files because
of the missing idls in dist/idl.
--HG--
extra : commitid : 7SoYbKdAeSQ
extra : rebase_source : 8fee8fdae0d04aed3906d00d33dd7b830f82d75f
extra : amend_source : 2a9a1086d409fbcd4386b7d589c8df50dcdf4b26
The current rule is only for "backend.RecursiveMakeBackend", but, with
the current default of generating both the RecursiveMake and FasterMake
backends, the command creates/refreshes both backends. This is, in fact,
how the FasterMake backend is refreshed in most cases.
Moreover, with an hybrid backends, the generated file is not
"backend.RecursiveMakeBackend" anymore, so we need a more generic way to
handle this.
Furthermore, it's not necessarily desirable for all backends to have a
dependency file to handle the dependencies to refresh the backend, so
generate a plain list instead. This has the side effect of making `mach
build-backend --diff` more readable for changes to that file.
Finally, make the backend.* files created like any other backend file,
such that its diff appears in the `mach build-backend --diff` output.
Install manifests are not empty in normal conditions, apart a few
exceptions where they are only used for a "magic" `rm -rf`.
However, we're going to introduce changes that will empty some of
the install manifests and make their work happen from a different
backend, in which case we don't want them to correspond to a `rm -rf`.
While it would be possible to move those things in the export tier, it
is still interesting to have reporting for them separately, especially
considering I expect things to gradually move from the other tiers to
this new one.
While not entirely obvious, the recurse_pre-export target doesn't imply
actual recursion as long as the RecursiveMake backend doesn't emit
traversal information for it, so nothing will actually happen on this
target, but the interesting part is that it runs, per the generic
config/recurse.mk rules for tiers, between BUILDSTATUS TIER_START and
BUILDSTATUS TIER_FINISH, so that all its dependencies are accounted
as being part of the pre-export tier.
This change allows specifying objdir-relative paths in EXPORTS to enable
exporting entries from GENERATED_FILES. Objdir paths in EXPORTS that are
not in GENERATED_FILES will raise an exception.
Example:
```
EXPORTS += ['!g.h', 'f.h']
GENERATED_FILES += ['g.h']
```
Given the implementation, this should also work for FINAL_TARGET_FILES,
FINAL_TARGET_PP_FILES, and TESTING_FILES, but those are not well-tested.
This patch also renames the install manifest for '_tests' to match the
directory name for convenience in some code I refactored.
--HG--
extra : commitid : CwayzXtxv1O
extra : rebase_source : 5fb6f461fc740da9bce14bbdbfabdfe618af8803
We have many unit tests in the tree for some small parts of the build system
pipeline, but we don't have anything that resembles an end to end test, and we
kind of rely on the resulting Firefox not being broken by our changes.
With the Faster make backend growing, I want to ensure it produces the same
thing as the recursive make backend, at least for the parts it supports.
This adds some kind of test that allows to check that.
The configure option has explicitly thrown an error for more than a year now,
and it happens that the remaining way to still forcefully use it has been
broken for more than 8 months.
Mike Hommey
Tom Prince
Chris Manchester
Ted Mielczarek
Mike Shal
Andrew Halberstadt
Sebastian Hengst
Gregory Szorc
Ehsan Akhgari
Ryan VanderMeulen
Dustin J. Mitchell
Birunthan Mohanathas
Steve Fink