The symbol-upload task currently downloads the symbols-full.zip artifact
from the build task and then uploads it to the symbol server. These zip
files can be very large (>1GB) so we spend a lot of time doing that.
Now that we're uploading to Tecken instead of Socorro, we can instead
just send the URL of the artifact to Tecken's upload API and ask it to
fetch that directly:
https://tecken.readthedocs.io/en/latest/upload.html#upload-by-download-url
This should make the symbol upload task a fair bit faster.
MozReview-Commit-ID: 8HcbgrWYT1O
--HG--
extra : rebase_source : 4e8f7a28c956befb3e291e8be4d41a2b6728e5cd
Bug 1256642 introduced magic at the emitter level to determine whether a
binary contains C++ sources and should be linked with the C compiler or
the C++ compiler.
Unfortunately, the Binary() moz.build template always adds C++ OS
libraries on Android (through STLPORT_LIBS), and C++ libraries on Linux
(stdc++compat).
The latter only ends up forcing every Binary() to be linked with the C++
linker, which is unfortunate, but doesn't cause much problems. The
former, however, involving OS libraries, the magic from bug 1256642
doesn't kick in, so we end up trying to link C++ OS libraries with the C
linker. Which ends up failing, because the libraries in STLPORT_LIBS
require -lm, which, while it's added by the C++ compiler when linking,
is not when the linkage is driven by the C compiler.
Because the fallible library, linked to all GeckoBinary()s is a C++
library, we still ended up linking with the C++ compiler on Android, so
this wasn't actually causing any problem... until I tried to remove that
fallible library in bug 1423803.
Anyways, the core problem is that moz.build evaluation is happening too
early to know whether any C++ sources are being linked together, so
there is no way the Binary() template can do the right thing. So this
change moves the logic to the emitter.
This also changes the type of STLPORT_LIBS to a list.
--HG--
extra : rebase_source : a70ddf7a132f94dc10e7e1db94ae80fb8d7a269f
This change makes upload-symbols tasks use run-task and the in-tree lint
image instead of the private upload-symbols image. A prior change changed
the script to get the token it needs from a Taskcluster secret, so it's no
longer necessary to use the private docker image containing the token.
MozReview-Commit-ID: 6QugVB4chE0
--HG--
extra : rebase_source : e13d29c2a88e055247da374cffa9ea153548749d
This change fixes symbol upload to use a token stored in the Taskcluster
secrets service instead of the token stored in the private Docker image.
Additionally, it changes the script to upload symbols to the Tecken staging
server when run on try so that the upload-symbols tasks can be tested on
try now. In the future there are plans to allow try tasks to upload symbols
to a separate storage area on the production Tecken instance.
MozReview-Commit-ID: BeZGiiwuGp8
--HG--
extra : rebase_source : ee4c680410822e94c3001d07242f69378703659f
This removes an unnecessary level of indirection by replacing all
nsStringGlue.h instances with just nsString.h.
--HG--
extra : rebase_source : 340989240af4018f3ebfd92826ae11b0cb46d019
This includes tests that cover both regular CFI stack walking as well as
pathological corner cases.
MozReview-Commit-ID: GDARnPSemyu
--HG--
extra : source : 1b65c0b41ac31f3645b2318b47072a1100c13183
We want symbolstore.py to fail, preferably loudly, if we can't find the
necessary tools, and throwing away errors here runs counter to that
goal. Dumper is a base class for Dumper_Win32, where we probably don't
have file(1), but Dumper_Win32 shouldn't be calling RunFileCommand.
This removes dead code using headlessClient and lastRunCrashID in crash
reporting. headlessClient is unconditional now. nsIXULRuntime.lastRunCrashID
is not used anymore so remove code for implementing it.
MozReview-Commit-ID: AU4bUeIx3O0
At present it is difficult to determine whether a crash ping is from a
shutdownkill or not. By including ipc_channel_error we will be able to figure
that out.
We also as a bonus get additional insight into ipc channel error types that
lead to crashes.
MozReview-Commit-ID: FepLsSS2tAI
--HG--
extra : rebase_source : 8c17bdd63f7fadb9829df63fe50c044a020b183e
This includes tests that cover both regular CFI stack walking as well as
pathological corner cases.
MozReview-Commit-ID: GDARnPSemyu
--HG--
extra : rebase_source : 99920e03174824020e4b80269c44f34b93b0364a
extra : source : 0560939928bb0f2fe019fa800fe8ee7663db4b8f
memory.h conflicts with a system header, so we have workarounds to
change include paths to work around this.
This is mostly a cherry-pick of this upstream commit:
8bb3d55af7
..but the patch was applied separately to toolkit/crashreporter/google-breakpad
and toolkit/crashreporter/breakpad-client since we've forked the latter,
and there's also one other fixup of a source file included.
MozReview-Commit-ID: HH92HZG7y9n
--HG--
rename : toolkit/crashreporter/google-breakpad/src/common/memory.h => toolkit/crashreporter/google-breakpad/src/common/memory_allocator.h
rename : toolkit/crashreporter/google-breakpad/src/common/memory_unittest.cc => toolkit/crashreporter/google-breakpad/src/common/memory_allocator_unittest.cc
extra : rebase_source : d321475099f000482689d6a6fb8629274ee19a65
extra : histedit_source : d526c27d952dbe73aee87e24701e2a862e1ca3d2
By using the PartialConfigEnvironment, the clients of buildconfig will
depend on config.statusd/ files instead of config.status directly.
Clients can access substs and defines using buildconfig.substs['FOO'] or
buildconfig.defines['BAR'], and then collect file-level dependencies for
make using buildconfig.get_dependencies(). All GENERATED_FILES rules
already make use of this because file_generate.py automatically includes
these dependencies (along with all python modules loaded).
As a result of this commit, re-running configure will no longer cause
the world to be rebuilt. Although config.status is updated, no build
steps use config.status directly and instead depend on values in
config.statusd/, which are written with FileAvoidWrite. Since those
files are not official targets according to the make backend, make won't
try to continually rebuild the backend when those files are out of date.
And since they are FileAvoidWrite, make will only re-run dependent steps
if the actual configure value has changed.
As a result of using JSON to load data from the config.statusd
directory, substs can be unicode (instead of a bare string type).
generate_certdata.py converts the subst manually to a string so the
value can be exported to the environment without issue on Windows.
Additionally, patching the buildconfig.substs dict no longer works, so
the unit-symbolstore.py test was modified to patch the underlying
buildconfig.substs._dict instead.
The other files that needed to be modified make use of all the defines
for the preprocessor. Those that are used during 'mach build' now use
buildconfig.defines['ALLDEFINES'], which maps to a special
FileAvoidWrite file generated for the PartialConfigEnvironment.
MozReview-Commit-ID: 2pJ4s3TVeS8
--HG--
extra : rebase_source : d6bb0208483f9f043e7be1b36907ca13243985f8
Unfortunately this also needs to be kept in Makefile.in to handle
other consumers of INCLUDES while we transition them.
MozReview-Commit-ID: 9OYlu6Jv1XZ
--HG--
extra : rebase_source : 719200501a93e836a03a64b5e1cd950a8f2e696a
The GPU process doesn't have the directory service enabled, so it can't find
a tmp dir to put its .extra files for crash reports. Even if we do enable the
directory service, we still don't get the correct "content process tmp dir" in
the GPU process, because the UUID baked into that folder is passed via the
preferences service, and that isn't initialized in the GPU process either.
Rather than unneccessarily initialize all this stuff in the GPU process just
to get one folder name, we can pass that folder name directly in the argv list.
See comments 12-19 on the bug for further discussion of the various solutions
attempted/explored.
MozReview-Commit-ID: 1sFg27hIe7S
--HG--
extra : rebase_source : 78eb600a58fed45800b9df8303cc4d6898d96ae9
The GPU process doesn't have the directory service enabled, so it can't find
a tmp dir to put its .extra files for crash reports. Even if we do enable the
directory service, we still don't get the correct "content process tmp dir" in
the GPU process, because the UUID baked into that folder is passed via the
preferences service, and that isn't initialized in the GPU process either.
Rather than unneccessarily initialize all this stuff in the GPU process just
to get one folder name, we can pass that folder name directly in the argv list.
See comments 12-19 on the bug for further discussion of the various solutions
attempted/explored.
MozReview-Commit-ID: 1sFg27hIe7S
--HG--
extra : rebase_source : 62ff819c5b03f642cd4b9af7c89c84b790397372
This modifies the minidump analyzer to only output the crashing thread's stack
when invoked. This reduces the raw size of the crash ping by 50% to 70%
depending on the program state when the crash occurred. The minidump-analyzer
can still produce the complete output when invoked with the '--full' option.
MozReview-Commit-ID: 8KaITWqfBKR
--HG--
extra : rebase_source : b71bd47ea1df4fa8fa3da013aef2612a073a42d2
symbolstore.py processes filenames in FILE lines of symbol files to encode
information about the source repository they came from, or to mark
known generated source files. It also reads the dist/include install
manifest so it can map header files from there back to their source locations.
These mappings were broken on Windows because symbolstore.py first passes
filenames into `FixFilenameCase`, which calls `GetFinalPathNameByHandleW`,
which breaks things in two ways:
1) It returns paths with an uppercase drive letter, and source paths from
elsewhere have a lowercase drive letter.
2) It resolves symlinks, and on Taskcluster Windows builds the whole build
is done within a symlinked directory so paths directly from the srcdir
and objdir won't match those canonicalized paths.
This patch adds a `normpath` function to symbolstore.py and moves the
contents of `FixFilenameCase` into it on Windows, and just makes it
an alias for `os.path.normpath` everywhere else. It then uses it everywhere
we deal with paths that will be compared against source file paths from symbol
files so that all paths are canonicalized the same and we can do simple
string matching from there.
Additionally, this patch adds a check to the functional test to verify
that header files from dist/include are correctly mapped to the source
repository. Unfortunately there is still not a test for generated files
because they only appear in the libxul symbol file, and dumping symbols
from libxul is too slow to invoke as part of a unit test.
MozReview-Commit-ID: Dx3z1BZcIvc
--HG--
extra : rebase_source : 80179bbea58a804344a56ef27f438ada76e7fe77
Bug 1360308 offloads IO operations from the main thread when we create paired minidumps.
This breaks the symmetry of paired minidumps: the thread stacks of the parent minidump
doesn't correspond to the thread stacks in the child minidumps and renders the parent
stack useless. This patch moves generation of the parent minidump back to the main
thread to keep the context of the parent process when creating paired minidumps. Child
minidump is still created asynchronously.
MozReview-Commit-ID: 9RmBAuXMPSX
Revert revision f760842b14a2, 051b765ca8f2 and 01125b5142e5 since the original
bug that we run out of TLS slots on Windows is no longer showing up after
firefox55. It should have been fixed elsewhere, very likely in the rust part.
MozReview-Commit-ID: 9j5hFSGT3OE
Now that builds are uploading generated source files to an S3 bucket,
symbolstore.py can alter the FILE lines in symbol files to record the
URLs where those generated source files can be found. We currently record
files from the hg repository as `hg:<repo>:<path>:<revision>`, so here we
record generated files as `s3:<bucket>:<path>:` and expect that Socorro
will map that to the S3 bucket in a sensible way.
This patch does not change source server indexing, which allows Microsoft
debuggers to fetch source files for a build. That will be handled in a
followup.
MozReview-Commit-ID: 1g14smF0fo8
--HG--
extra : rebase_source : a5b42251278e6ecc4d57e374c423738de325f8e5
extra : source : 7781a37a4db0378b06ef3ad377e18be37e63ad83
Now that builds are uploading generated source files to an S3 bucket,
symbolstore.py can alter the FILE lines in symbol files to record the
URLs where those generated source files can be found. We currently record
files from the hg repository as `hg:<repo>:<path>:<revision>`, so here we
record generated files as `s3:<bucket>:<path>:` and expect that Socorro
will map that to the S3 bucket in a sensible way.
This patch does not change source server indexing, which allows Microsoft
debuggers to fetch source files for a build. That will be handled in a
followup.
MozReview-Commit-ID: 1g14smF0fo8
--HG--
extra : rebase_source : 78344e2435cc2379d4b202b71851d1112236066b
This mechanically replaces nsILocalFile with nsIFile in
*.js, *.jsm, *.sjs, *.html, *.xul, *.xml, and *.py.
MozReview-Commit-ID: 4ecl3RZhOwC
--HG--
extra : rebase_source : 412880ea27766118c38498d021331a3df6bccc70
The 'src' subdir needs to be part of the path *after*
the blob/commit_id section of the url, so we need to
no strip it from the prefix when we match.
MozReview-Commit-ID: 9HA3a7d8kh4
--HG--
extra : rebase_source : 6a558c06085844cd7b664c1c65a6e5023ecbb423
This option was added ~decade ago; AFAICT from bug archaeology, the
option was added to prevent our servers from being overwhelmed.
Somewhere over the years, however, we obtained more capable servers and
the option disappeared from mozconfigs. It seems moderately unlikely
that we'll have a need for this option again, and we could reintroduce
this patch very easily in any event. Let's go ahead and remove it.
It's just a complex wrapper for free(), or equivalent function. (In practice,
all the uses end up in free().)
--HG--
extra : rebase_source : 247ea8458aa57319bd1c8366115a9b4f39ed5a33
Install crash reporter's panic hook in child processes (and also delay the main
process installation until we know crash reporter is enabled).
When collecting child crash annotations, read the Rust panic message if it
exists.
MozReview-Commit-ID: Gfp2E8IHjw8
--HG--
extra : rebase_source : 9e261f399e7c54fc262a1500cf2514ddd1012406
We were prefix-matching the rust srcdir when hyperlinking
symbols, but then appending the relative source path to
the top level repo url, resulting in broken links.
Instead, link to the srcdir url at github.
MozReview-Commit-ID: 33tSMM96Vie
--HG--
extra : rebase_source : 13d5638d4029e40077c0ba8de34a64d3ff9e92be
It's silly to use prmem.h within Firefox code given that in our configuration
its functions are just wrappers for malloc() et al. (Indeed, in some places we
mix PR_Malloc() with free(), or malloc() with PR_Free().)
This patch removes all uses, except for the places where we need to use
PR_Free() to free something allocated by another NSPR function; in those cases
I've added a comment explaining which function did the allocation.
--HG--
extra : rebase_source : 0f781bca68b5bf3c4c191e09e277dfc8becffa09
This gives us source file names with repository info in our generated
symbol files, so that crash reports on crash-stats can link to the
correct source files for files from the Rust standard library.
I've hardcoded the source paths that the Rust project uses, which is
not my favorite thing, but there's no simple way to get this information
otherwise.
MozReview-Commit-ID: 6SeaMqH8xfc
--HG--
extra : rebase_source : 227a52db42b3f86378d744e68b4d88d2d8ba9757
This moves the I/O operations of writing minidumps in CrashReporter::CreateMinidumpsAndPair()
off the main thread and use callbacks to notify the completion of the operations.
MozReview-Commit-ID: 9wpTDDUp9GN
This deduplicates/merges app memory regions so that we don't generate overlapping
memory regions in the minidump. We need to do this because older version of
dbghelp.dll (e.g. that on Windows 7) doesn't handle overlaps in its memory list,
which causes xpcshell bustages.
The test bustages are fixed by sorting and merging app memory regions before
adding to the input to the MinidumpWriteDump callback. Memory regions on the thread
stack is also excluded because thread stack memory is included in the minidump
by default.
MozReview-Commit-ID: 541vzG0b0kC
This patch is the core part of including heap regions using the registers of
the crashing context to be used by in-process and out-of-process crash
generations. It works as follows:
When a crash happens, the ExceptionHandler or CrashGenerationServer uses the register
values in the crash contex as candidates for heap regions. If a value "looks like"
a pointer to some data on the heap, we register a region around the value using a
preallocated AppMemory instance. Then in the "MemoryCallback" type of MinidumpCallback,
we register the memory regions for inclusion in the minidump.
MozReview-Commit-ID: BEik3lRJJkj
This moves type definitions and minidump callback to minidump_callback.{h|cpp},
which will be used by google_breakpad::ExceptionHandler for in-process crash
generation, and by googld_breakpad::CrashGenerationServer for out-of-process
crash generation.
MozReview-Commit-ID: AMsQHSUTYNx
We will do the same thing for all versions of this DLL that ship
on the versions of Windows that we support, so this currently
only serves as a way to slow down startup.
This also removes the TelemetrySessionId field from the whitelist because that
field is already manually stripped and moved into the ping's root node.
Putting it in the whitelist is redundant.
MozReview-Commit-ID: B3WAyCSHoyR
--HG--
extra : rebase_source : 9729399a1e1c45cf1646498379d68684985134ce
This adds the RemoteType annotation to a content crash report so that we can
distinguish between content processes that crashed while running remote, local
or extension code. The annotation is passed along the others to Socorro by the
crashreporter and is also whitelisted for inclusion in the crash ping.
MozReview-Commit-ID: 4avo0IWfMGf
--HG--
extra : rebase_source : 8d03f7e166b5762a5ce7cab13c2101302b4f1d2f
This patch removes the C++ code used to run the minidump analyzer when a
content process crashes, and replaces it with JS code within the CrashService
object. This removes the need for a separate shutdown blocker in C++ code and
allows end-to-end testing of the crash service functionality. Additionally
the exception handler code can be simplified since it's now only used to run
the crash reporter client.
The test added to test_crash_service.js covers computing the minidump SHA256
hash (bug 1322611) and of the minidump analyzer itself (bug 1280477).
MozReview-Commit-ID: LO5w839NHev
Detect the NDK major/minor version numbers, and feed that to Breakpad.
For AArch64, some Breakpad headers try to workaround NDK oddities by
checking the ANDROID_NDK_MAJOR_VERSION and ANDROID_NDK_MINOR_VERSION
macros.
Ted Mielczarek
Tom Prince
Dorel Luca
Mike Hommey
Gabriele Svelto
Sylvestre Ledru
Adam Gashlin
Cosmin Sabou
Eric Rahm
Marco Castelluccio
Jan Beich
shindli
Mark Banner
Dan Banner
Andreea Pavel
Dan Horák
Carl Corcoran
Nathan Froyd
Cervantes Yu
Chris H-C
Csoregi Natalia
Chris Manchester
Stephen A Pohl
David Major
Mike Shal
Chris Peterson
Kartikaya Gupta
Sebastian Hengst
Tom Ritter
Emilio Cobos Álvarez
Wes Kocher
Phil Ringnalda
Masatoshi Kimura
Rajesh Kathiriya
Ralph Giles
Nicholas Nethercote
J. Ryan Stinnett
Brad Werth
Perry Jiang
Bill McCloskey
Carsten "Tomcat" Book
Florian Quèze
Ehsan Akhgari
Markus Stange
Jim Chen