This patch makes it so we write the sccache log directly to the artifact
directory, so that it will be uploaded even if the build fails. It also makes
the log slightly more verbose. Both of these should help with diagnosing
sccache failures in CI.
The sccache log will no longer be explicitly gzip compressed, but some
Taskcluster client implementations will store it as gzip compressed.
Differential Revision: https://phabricator.services.mozilla.com/D6187
--HG--
extra : moz-landing-system : lando
This patch moves the logic for include/exclude/compress from the makefile into
symbols_archive.py, and adds an additional check to the full-archive case to
allow the inclusion of test symbols in try builds.
--HG--
extra : amend_source : 306419aae90a1935ced49fec89112df05142501c
Now that we've generated an order file of the first N functions invoked
during startup, let's tell the linker about said functions so it can
cluster them appropriately.
test_build.py fails on local builds with messages like:
make[4]: Entering directory '/tmp/tmp65BjCH'
make[4]: *** No rule to make target 'buildid.h'. Stop.
make[4]: Leaving directory '/tmp/tmp65BjCH'
/home/froydnj/src/gecko-dev.git/config/faster/rules.mk:76: recipe for target '/tmp/tmp65BjCH/buildid.h' failed
make[3]: *** [/tmp/tmp65BjCH/buildid.h] Error 2
make[3]: *** Waiting for unfinished jobs....
make[4]: Entering directory '/tmp/tmp65BjCH'
make[4]: *** No rule to make target 'source-repo.h'. Stop.
make[4]: Leaving directory '/tmp/tmp65BjCH'
/home/froydnj/src/gecko-dev.git/config/faster/rules.mk:76: recipe for target '/tmp/tmp65BjCH/source-repo.h' failed
make[3]: *** [/tmp/tmp65BjCH/source-repo.h] Error 2
make[3]: Leaving directory '/tmp/tmp65BjCH/faster'
Makefile:155: recipe for target 'faster' failed
The tests pass in automation, however, because automation always defines
the rules for buildid.h and source-repo.h in the toplevel Makefile.in.
For local builds, however, those rules are not defined to avoid build
churn. Let's ensure that the necessary rules are defined during testing
as well.
Tup can have a lot of unnecessary log lines, particularly on a fresh
build when it is scanning directories for the first time. Turning on
--quiet when MOZ_AUTOMATION is set will help trim down the log size
significantly.
MozReview-Commit-ID: AaRppurkXsA
--HG--
extra : rebase_source : dca18a0e3657853eacfc6cdbb935998cadb339ce
For automation builds, force buildid.h to be regenerated, and always
#include buildid.h in nsAppRunner.cpp. This will eliminate the mismatch at the
cost of re-linking libxul on every build.
For developer builds, always #include buildid.h in nsAppRunner.cpp, but do not
force buildid.h to be re-generated for incremental builds. This means we will
never have a mismatch between platform and application buildids, although
subsequent builds are no longer guaranteed to have a new buildid.
MozReview-Commit-ID: 19hvu8AQXgN
--HG--
extra : rebase_source : 9f4fd063f89b9b2165d5889fdaa8f83104c4411c
For automation builds, force buildid.h to be regenerated, and always
#include buildid.h in nsAppRunner.cpp. This will eliminate the mismatch at the
cost of re-linking libxul on every build.
For developer builds, always #include buildid.h in nsAppRunner.cpp, but do not
force buildid.h to be re-generated for incremental builds. This means we will
never have a mismatch between platform and application buildids, although
subsequent builds are no longer guaranteed to have a new buildid.
MozReview-Commit-ID: 19hvu8AQXgN
--HG--
extra : rebase_source : 181b53c2cc47526bd632a0931ba6e65fc74179ef
It is a one-liner and doesn't need to exist in a separate make file.
MozReview-Commit-ID: 5w3eLWD0CuB
--HG--
extra : rebase_source : 15ebb63156f8b4e8d3288dfaf0d9ec2c328b5914
Previously, client.mk made the decision of whether to perform a PGO
build. This required passing around MOZ_PGO and invoking a separate
make target if this variable was set.
In this commit, we move this logic to Makefile.in. We employ a special
mechanism in rules.mk to override the default make target so
`make` evaluates "profiledbuild" if MOZ_PGO is set. This also
required using an explicit target for $(MAKE) invocations inside
the "profiledbuild" rule to avoid infinite recursion.
MozReview-Commit-ID: 8sHiVspMisM
--HG--
extra : rebase_source : 76c8a0c592015802f2b9ad52fe2001012a4611f6
extra : source : 24e28b37fbc0ad55ee449699e09c62042af41b55
client.mk is going away. This commit moves the PGO build target out of
it.
We move the code to Makefile.in. This likely isn't its final
home, as we'll want to eliminate Makefile.in at some point. However,
it is easier to move things to the root Makefile.in than other
alternatives. Keep in mind that the logic needs access to primitives
like BUILDSTATUS and variables from mk_add_options, which the
Makefile.in mechanism has easy access to.
MozReview-Commit-ID: 4C43lBhZz4l
--HG--
extra : rebase_source : 0ff1e95dcf8cb0082b019713db0cb7f47f0bccb8
extra : source : 0d2083b8edc43fc13c77f71199e0fcaddb32fcfb
client.mk hasn't been well-supported for years. Stop recommending it.
MozReview-Commit-ID: KWFhAQPIXWu
--HG--
extra : rebase_source : dafb307d82f1f0a3141d077102fd2fe21add65b4
The --track flag provides a more accurate accounting of what files were
installed by the manifest, so they can be appropriately removed. For
example, test files are now removed from _tests if an entry in a test
file is deleted.
The --no-remove flag is removed as an alternative, and the --track flag
is now mandatory.
MozReview-Commit-ID: Wiup4Gzwkb
--HG--
extra : rebase_source : 4a44c7fe066ba9b5f1e37ec682464f7f4f6cb2cf
The SDK installation code was removed in bug 1333826.
MozReview-Commit-ID: 3h63nFSKOfi
--HG--
extra : rebase_source : 4a99f56e62c749e8991831f1a86cb4d98a58d31e
This change makes us upload an `$(PKG_BASENAME).generated-files.tar.gz` archive
alongside other build artifacts which contains all the generated source files
from the build. A change after this will introduce an `upload-generated-sources`
task to take this artifact and upload the individual files to an S3 bucket.
This will be used to provide links to generated source files when they appear
in stack traces in crash reports.
MozReview-Commit-ID: 6yQAdlZ5q3O
--HG--
extra : rebase_source : d92fb17ae737d1360e9724997f6688e29bedef12
extra : source : 14d18d7cf454c4c3d0f6d49d1d01660e06e4be4b
This change makes us upload an `$(PKG_BASENAME).generated-files.tar.gz` archive
alongside other build artifacts which contains all the generated source files
from the build. A change after this will introduce an `upload-generated-sources`
task to take this artifact and upload the individual files to an S3 bucket.
This will be used to provide links to generated source files when they appear
in stack traces in crash reports.
MozReview-Commit-ID: 6yQAdlZ5q3O
--HG--
extra : rebase_source : 3f6ef734c062e0f5e9c2ca433ffad51fdf14b1ad
Note this will only enable it on try, autoland and mozilla-inbound,
which are the only branches where sccache are enabled at the moment.
Enabling on more builds (or not) is the subject of bug 1373460.
Also note that bug 1181040, that ensured PGO builds weren't using
sccache mentions that back then, link times went up when using sccache
(with -Z7) vs. without, but that was presumably with MSVC 2013. Try
suggests link times are the same using sccache now (still using -Z7,
pending bug 1318370).
--HG--
extra : rebase_source : 9f9d87714f3c841b790eb7d692ea0968b1755b70
When investigating why a particular build got a particularly high number
of cache misses, it is useful to know exactly *what* had cache misses,
and you can't get the missing information after the fact.
We used to have some stats stored in a log file uploaded as artifact
with the old python sccache per bug 1005495 (and before that inline in
the build log), and we actually still have remains of that in the form
of a EXTRA_UPLOAD_FILES in build/mozconfig.cache.
Let's reuse that, and get some useful logs from sccache. Ideally sccache
would give better logs, but those are sufficient for now. We'll figure
things out eventually. https://github.com/mozilla/sccache/issues/151
--HG--
extra : rebase_source : a2e7093697a5b720ab28f155560211d6c0a4347e
Thunderbird is still using MOZ_PKG_PRETTYNAMES which has spaces in the package
name, which gets mangled by abspath.
MozReview-Commit-ID: 23ngx6gIESI
--HG--
extra : rebase_source : 9038f295e582bec5872898a5e8f052bfacf8f85a
This commit makes sccache dump JSON stats at the end of the build, and then
reads them in `BuildScript.generate_build_stats` and adds them to the
build_metrics we submit to Perfherder. The stats dumping is done in
Makefile.in where we currently dump verbose sccache stats because sccache
doesn't persist stats to disk right now and it will also shut down its server
process after 5 minutes, so when the post-build automation steps take more
than 5 minutes the server shuts down and the stats are lost.
Currently it's collecting:
* Cache hit rate
* Cache write errors
* Non-cacheable requests (compiler invocations that sccache can't cache)
We can always grow this list later.
MozReview-Commit-ID: J9CwU7XB05I
--HG--
extra : rebase_source : 084b09c3b0621330ac331a99b1bca9a15cf833b7
This commit moves symbol dumping to the compile tier, to be run via "syms"
targets. Tracking files are used for the sake of incremental builds, because
dump_syms may genearate multiple outputs whose paths are not known ahead of
time.
Minimal changes to symbolstore.py are made here. More extensive
simplifications will be made in a future commit on the basis of symbolstore.py
handling one file at a time.
MozReview-Commit-ID: 3mOP8A6Y7iM
--HG--
extra : rebase_source : bfe97afcdfc05b9e79f01577701c83e8b00eb4e9
Ted Mielczarek
Gabriele Svelto
Mike Hommey
arthur.iakab
David Major
Andreea Pavel
Chris AtLee
Aaron Klotz
Nathan Froyd
Chris Manchester
Ciure Andrei
Nathan Froyd
Mike Shal
Gurzau Raul
Tom Prince
Dorel Luca
Gregory Szorc
Wes Kocher