It turns out sometimes (in the LTO+CFI case at least) Assertions.h
will not be present in the opt build, presumably because it was optimized
out.
MozReview-Commit-ID: GB3GIoSdIUK
The crash reporter symbol files are the easiest cross-platform way to
find static initializers. While some types of static initializers (e.g.
__attribute__(constructor) functions) don't appear there in a notable
way, the static initializers we do care the most about for tracking do
(static initializers from C++ globals). As a matter of fact, there is
only a difference of 2 compared to the currently reported count of 125
on a linux64 build, so this is a good enough approximation. And allows
us to easily track the count on Android, OSX and Windows builds, which
we currently don't do.
The tricky part is that the symbol files are in
dist/crashreporter-symbols/$lib/$fileid/$lib.sym, and $fileid is hard to
figure out. There is a `fileid` tool in testing/tools, but it is a
target binary, meaning it's not available on cross builds (OSX,
Android).
So the simplest is just to gather the data while creating the symbol
files, which unfortunately requires to go through some hoops to make it
happen for just the files we care about.
--HG--
extra : rebase_source : 458fed1ffd6f9294eefef61f10ff7a284af0d986
The symbol server recently gained the ability to upload symbols from try
to a different prefix in the symbols bucket. If we change the token stored
in the Taskcluster secret `project/releng/gecko/build/level-1/gecko-symbol-upload`
to one that has only "upload try symbols" permissions then we can upload
symbols from try server builds directly to symbols.mo.
MozReview-Commit-ID: HjQclKKXbA3
--HG--
extra : rebase_source : 8afed0bf52565bad513c30a7d5de274b356d6d84
Historically we built all our binaries in directories in the objdir, then
symlinked them into dist/bin. Some binaries needed to be copied instead
so that certain relative path lookups work properly, so we resorted to
sprinkling `NSDISTMODE=copy` around Makefiles.
This change makes it so we build PROGRAMs (not any other sort of targets)
directly in dist/bin instead. We could do the same for our other targets
with a little more work.
There were several places in the tree that were copying built binaries to
some other place and needed fixup to match the new location of binaries.
On Windows pdb files are left in the objdir where the program was
originally linked. symbolstore.py needs to locate the pdb file both to
determine whether it should dump symbols for a binary and also to copy
the pdb file into the symbol package. We fix this by simply looking for
the pdb file in the current working directory if it isn't present next
to the binary, which matches how we invoke symbolstore.py.
MozReview-Commit-ID: 8TOD1uTXD5e
For consistency with when MOZ_SOURCE_CHANGESET is set, and because while
slim, there is a chance of conflict with short forms that don't exist
with the full form that could bite us in the long run.
--HG--
extra : rebase_source : fd7057ec58c6267f3f971a96fdf2854f2cd3d89b
Historically we built all our binaries in directories in the objdir, then
symlinked them into dist/bin. Some binaries needed to be copied instead
so that certain relative path lookups work properly, so we resorted to
sprinkling `NSDISTMODE=copy` around Makefiles.
This change makes it so we build PROGRAMs (not any other sort of targets)
directly in dist/bin instead. We could do the same for our other targets
with a little more work.
There were several places in the tree that were copying built binaries to
some other place and needed fixup to match the new location of binaries.
On Windows pdb files are left in the objdir where the program was
originally linked. symbolstore.py needs to locate the pdb file both to
determine whether it should dump symbols for a binary and also to copy
the pdb file into the symbol package. We fix this by simply looking for
the pdb file in the current working directory if it isn't present next
to the binary, which matches how we invoke symbolstore.py.
MozReview-Commit-ID: 8TOD1uTXD5e
--HG--
extra : rebase_source : 9140be949b206bb595d9188ce7e8357347ecd9a9
The symbol-upload task currently downloads the symbols-full.zip artifact
from the build task and then uploads it to the symbol server. These zip
files can be very large (>1GB) so we spend a lot of time doing that.
Now that we're uploading to Tecken instead of Socorro, we can instead
just send the URL of the artifact to Tecken's upload API and ask it to
fetch that directly:
https://tecken.readthedocs.io/en/latest/upload.html#upload-by-download-url
This should make the symbol upload task a fair bit faster.
MozReview-Commit-ID: 8HcbgrWYT1O
--HG--
extra : rebase_source : 4e8f7a28c956befb3e291e8be4d41a2b6728e5cd
This change makes upload-symbols tasks use run-task and the in-tree lint
image instead of the private upload-symbols image. A prior change changed
the script to get the token it needs from a Taskcluster secret, so it's no
longer necessary to use the private docker image containing the token.
MozReview-Commit-ID: 6QugVB4chE0
--HG--
extra : rebase_source : e13d29c2a88e055247da374cffa9ea153548749d
This change fixes symbol upload to use a token stored in the Taskcluster
secrets service instead of the token stored in the private Docker image.
Additionally, it changes the script to upload symbols to the Tecken staging
server when run on try so that the upload-symbols tasks can be tested on
try now. In the future there are plans to allow try tasks to upload symbols
to a separate storage area on the production Tecken instance.
MozReview-Commit-ID: BeZGiiwuGp8
--HG--
extra : rebase_source : ee4c680410822e94c3001d07242f69378703659f
We want symbolstore.py to fail, preferably loudly, if we can't find the
necessary tools, and throwing away errors here runs counter to that
goal. Dumper is a base class for Dumper_Win32, where we probably don't
have file(1), but Dumper_Win32 shouldn't be calling RunFileCommand.
By using the PartialConfigEnvironment, the clients of buildconfig will
depend on config.statusd/ files instead of config.status directly.
Clients can access substs and defines using buildconfig.substs['FOO'] or
buildconfig.defines['BAR'], and then collect file-level dependencies for
make using buildconfig.get_dependencies(). All GENERATED_FILES rules
already make use of this because file_generate.py automatically includes
these dependencies (along with all python modules loaded).
As a result of this commit, re-running configure will no longer cause
the world to be rebuilt. Although config.status is updated, no build
steps use config.status directly and instead depend on values in
config.statusd/, which are written with FileAvoidWrite. Since those
files are not official targets according to the make backend, make won't
try to continually rebuild the backend when those files are out of date.
And since they are FileAvoidWrite, make will only re-run dependent steps
if the actual configure value has changed.
As a result of using JSON to load data from the config.statusd
directory, substs can be unicode (instead of a bare string type).
generate_certdata.py converts the subst manually to a string so the
value can be exported to the environment without issue on Windows.
Additionally, patching the buildconfig.substs dict no longer works, so
the unit-symbolstore.py test was modified to patch the underlying
buildconfig.substs._dict instead.
The other files that needed to be modified make use of all the defines
for the preprocessor. Those that are used during 'mach build' now use
buildconfig.defines['ALLDEFINES'], which maps to a special
FileAvoidWrite file generated for the PartialConfigEnvironment.
MozReview-Commit-ID: 2pJ4s3TVeS8
--HG--
extra : rebase_source : d6bb0208483f9f043e7be1b36907ca13243985f8
symbolstore.py processes filenames in FILE lines of symbol files to encode
information about the source repository they came from, or to mark
known generated source files. It also reads the dist/include install
manifest so it can map header files from there back to their source locations.
These mappings were broken on Windows because symbolstore.py first passes
filenames into `FixFilenameCase`, which calls `GetFinalPathNameByHandleW`,
which breaks things in two ways:
1) It returns paths with an uppercase drive letter, and source paths from
elsewhere have a lowercase drive letter.
2) It resolves symlinks, and on Taskcluster Windows builds the whole build
is done within a symlinked directory so paths directly from the srcdir
and objdir won't match those canonicalized paths.
This patch adds a `normpath` function to symbolstore.py and moves the
contents of `FixFilenameCase` into it on Windows, and just makes it
an alias for `os.path.normpath` everywhere else. It then uses it everywhere
we deal with paths that will be compared against source file paths from symbol
files so that all paths are canonicalized the same and we can do simple
string matching from there.
Additionally, this patch adds a check to the functional test to verify
that header files from dist/include are correctly mapped to the source
repository. Unfortunately there is still not a test for generated files
because they only appear in the libxul symbol file, and dumping symbols
from libxul is too slow to invoke as part of a unit test.
MozReview-Commit-ID: Dx3z1BZcIvc
--HG--
extra : rebase_source : 80179bbea58a804344a56ef27f438ada76e7fe77
Now that builds are uploading generated source files to an S3 bucket,
symbolstore.py can alter the FILE lines in symbol files to record the
URLs where those generated source files can be found. We currently record
files from the hg repository as `hg:<repo>:<path>:<revision>`, so here we
record generated files as `s3:<bucket>:<path>:` and expect that Socorro
will map that to the S3 bucket in a sensible way.
This patch does not change source server indexing, which allows Microsoft
debuggers to fetch source files for a build. That will be handled in a
followup.
MozReview-Commit-ID: 1g14smF0fo8
--HG--
extra : rebase_source : a5b42251278e6ecc4d57e374c423738de325f8e5
extra : source : 7781a37a4db0378b06ef3ad377e18be37e63ad83
Now that builds are uploading generated source files to an S3 bucket,
symbolstore.py can alter the FILE lines in symbol files to record the
URLs where those generated source files can be found. We currently record
files from the hg repository as `hg:<repo>:<path>:<revision>`, so here we
record generated files as `s3:<bucket>:<path>:` and expect that Socorro
will map that to the S3 bucket in a sensible way.
This patch does not change source server indexing, which allows Microsoft
debuggers to fetch source files for a build. That will be handled in a
followup.
MozReview-Commit-ID: 1g14smF0fo8
--HG--
extra : rebase_source : 78344e2435cc2379d4b202b71851d1112236066b
The 'src' subdir needs to be part of the path *after*
the blob/commit_id section of the url, so we need to
no strip it from the prefix when we match.
MozReview-Commit-ID: 9HA3a7d8kh4
--HG--
extra : rebase_source : 6a558c06085844cd7b664c1c65a6e5023ecbb423
We were prefix-matching the rust srcdir when hyperlinking
symbols, but then appending the relative source path to
the top level repo url, resulting in broken links.
Instead, link to the srcdir url at github.
MozReview-Commit-ID: 33tSMM96Vie
--HG--
extra : rebase_source : 13d5638d4029e40077c0ba8de34a64d3ff9e92be
This gives us source file names with repository info in our generated
symbol files, so that crash reports on crash-stats can link to the
correct source files for files from the Rust standard library.
I've hardcoded the source paths that the Rust project uses, which is
not my favorite thing, but there's no simple way to get this information
otherwise.
MozReview-Commit-ID: 6SeaMqH8xfc
--HG--
extra : rebase_source : 227a52db42b3f86378d744e68b4d88d2d8ba9757
This removes handling of dumping symbols in parallel from symbolstore.py
and updates unit tests.
A prior commit made symbolstore.py handle a single file at a time, leaving
concurrency to be handled by make, so this is no longer needed.
MozReview-Commit-ID: C7IHdVHHjRH
--HG--
extra : rebase_source : 4e90ca5d5f64fe0b1cd29a08d5c262dc0fdfcc33
This commit moves symbol dumping to the compile tier, to be run via "syms"
targets. Tracking files are used for the sake of incremental builds, because
dump_syms may genearate multiple outputs whose paths are not known ahead of
time.
Minimal changes to symbolstore.py are made here. More extensive
simplifications will be made in a future commit on the basis of symbolstore.py
handling one file at a time.
MozReview-Commit-ID: 3mOP8A6Y7iM
--HG--
extra : rebase_source : bfe97afcdfc05b9e79f01577701c83e8b00eb4e9
This will be more convenient when we dump symbols from the compile tier.
MozReview-Commit-ID: Ltjq8ai5j0m
--HG--
extra : rebase_source : bcb059879b1dbadb2e84caa41247fd129629eb37
It turns out that running makecab to compress PDB files takes a significant
amount of time in the buildsymbols step. I wrote an implementation of
makecab in Rust that implements only the subset of features we use and
it's significantly faster:
https://github.com/luser/rust-makecab
This patch adds a makecab check to moz.configure, adds a release build of
the makecab binary to the Windows tooltool manifests, points the build at
it from mozconfig.win-common, and changes symbolstore.py to use MAKECAB
from substs instead of calling `makecab.exe` directly.
MozReview-Commit-ID: 76FHLIZFCXS
--HG--
extra : rebase_source : af4cf2e4db4607ec9329b2811cc0175d3e113b66
In some corner cases, there can be more than one entry for
nsBrowserApp.cpp for the firefox executable.
--HG--
extra : rebase_source : c0ff78aad7f43ed60cb0ac7413a01ba005e49dbc
If the srcdir is in a path containing a symlink on Windows, when
`FixFilenameCase` calls `GetFinalPathNameByHandleW` to normalize the case of
a source file it will get a path to the file with the symlink resolved.
This breaks our "is this file in the source repository" check.
This patch makes the code call `FixFilenameCase` for any srcdir arguments
that are passed to the script, so any symlinks will be resolved there
and the prefix matching will work.
MozReview-Commit-ID: 2UibW9XYWoK
--HG--
extra : rebase_source : 199a36f6109693ec72a9661a02147aedbbc35158
Ionut Goldan
Tom Ritter
Mike Hommey
Ted Mielczarek
Gurzau Raul
Tom Prince
Noemi Erli
Dorel Luca
Nathan Froyd
David Major
Mike Shal
Wes Kocher
Ralph Giles
Chris Manchester