Expand `profiler_is_locked_on_current_thread()` to also return true if the ProfileBufferGlobalController mutex is locked on the current thread.
This will prevent some profiler-re-entrant operations (like native allocation markers) from running.
In particular, this removes potential deadlocks similar to the one found in bug 1671403:
- SamplerThread: In the sampling loop, lock the main profiler mutex, run a local update, which attempts to lock the ProfileBufferGlobalController mutex.
- ProfilerChild thread: While processing an IPC message with an update, lock the ProfileBufferGlobalController mutex, then resolve the update, which records a native allocation with a backtrace that attempts to lock the main profiler mutex.
With this patch, the native allocation won't record a marker while the ProfileBufferGlobalController mutex is locked.
Differential Revision: https://phabricator.services.mozilla.com/D96970
Expand `profiler_is_locked_on_current_thread()` to also return true if the ProfilerChild mutex is locked on the current thread.
This will prevent some profiler-re-entrant operations (like native allocation markers) from running.
In particular, this removes the potential deadlock found in bug 1671403:
- SamplerThread: In the sampling loop, lock the main profiler mutex, run a ProfilerChild update, which attempts to lock the ProfilerChild mutex.
- ProfilerChild thread: While processing an IPC message with an update, lock the ProfilerChild mutex, then resolve the update, which records a native allocation with a backtrace that attempts to lock the main profiler mutex.
With this patch, the native allocation won't record a marker while the ProfilerChild mutex is locked.
Differential Revision: https://phabricator.services.mozilla.com/D96969
Some markers (e.g., GC major/minor/slice) need to splice JSON strings in their data.
So now, instead of JSONWriter, StreamJSONMarkerData functions will be given a mozilla::baseprofiler::SpliceableJSONWriter.
Differential Revision: https://phabricator.services.mozilla.com/D95511
Instead of a repeating timeout of only twice the parent's serialization time + 1s, use that double parent time and multiply it by the number of children, and add the number of seconds from the about:config preference "devtools.performance.recording.child.timeout_s" (still 1s by default).
Differential Revision: https://phabricator.services.mozilla.com/D94955
The baseprofiler is required to be in place first for features that are only
implemented in the baseprofiler, such as the MarkerThreadId::MainThread().
This patch adds the initialization, and upates the ProfilerIOInterposeObserver
to use baseprofiler-only features.
The issue before, was that the base profiler was not initialized, and the
mozilla::baseprofiler::profiler_main_thread_id() was incorrectly reporting that
the main thread index was 0.
No tests were changed in this patch, but it does have coverage with:
tools/profiler/tests/xpcshell/test_feature_fileioall.js
This test would fail with just the interposer changes.
Differential Revision: https://phabricator.services.mozilla.com/D94839
This commit uses the new Markers 2.0 API for FileIO Markers. I had to
create another option for the MarkerStack class in order to conditionally
capture a backtrace inside of the Macro. Otherwise the macro invocation
failed.
Differential Revision: https://phabricator.services.mozilla.com/D93848
This commit uses the new Markers 2.0 API for FileIO Markers. I had to
create another option for the MarkerStack class in order to conditionally
capture a backtrace inside of the Macro. Otherwise the macro invocation
failed.
Differential Revision: https://phabricator.services.mozilla.com/D93848
In most situations, JSONWriter users already know string lengths (either directly, or through `nsCString` and friends), so we should keep this information through JSONWriter and not recompute it again.
This also allows using JSONWriter with sub-strings (e.g., from a bigger buffer), without having to create null-terminated strings.
Public JSONWriter functions have overloads that accept literal strings.
Differential Revision: https://phabricator.services.mozilla.com/D86192
In most calls to `SpliceableJSONWriter::Splice(const char*)`:
- The data comes from a `ChunkedJSONWriteFunc` and is copied to a new buffer, which is then copied again through `Write()`. Instead we can copy the data directly from the `ChunkedJSONWriteFunc`; and this is a nice complement to `TakeAndSplice()` below.
- Or the length is already known, so we can pass it to a new `Splice(const char*, size_t)`, which forwards it to `Write(const char*, size_t)`, saving one `strlen` call.
Differential Revision: https://phabricator.services.mozilla.com/D87703
`SpliceableChunkedJSONWriter::ChunkedWriteFunc` returns a `ChunkedJSONWriteFunc*`, which is never null and is either used to:
1. Copy data.
2. Or take ownership of the chunks.
In the first case, `ChunkedWriteFunc()` now returns a `const ChunkedJSONWriteFunc&` (notice "const &"), so only const members may be used to copy the data.
In the second case, a new function `TakeChunkedWriteFunc()` returns `ChunkedJSONWriteFunc&&` (notice "&&"), so it's clear that its chunks can be taken away. Some `DEBUG` assertions help ensure that it's not used anymore after that.
`TakeAndSplice()` now takes a `ChunkedJSONWriteFunc&&`.
All callers have been updated to the more appropriate functions.
Differential Revision: https://phabricator.services.mozilla.com/D87702
SpliceableChunkedJSONWriter::WriteFunc was hiding base-class non-virtual JSONWriter::WriteFunc(), which made it less than ideal (for me) to reason with.
Also made a few subclasses final, to help with possible devirtualization.
Differential Revision: https://phabricator.services.mozilla.com/D86505
The main change is removing ProfileJSONWriter.cpp, making ProfileJSONWriter.h point at BaseProfileJSONWriter.h, and exposing `mozilla::baseprofiler::` classes in the top namespace as expected by users of ProfileJSONWriter.h (to minimize changes).
These two headers are now always present in the "mozilla" include directory, independent of MOZ_GECKO_PROFILER settings.
The rest is just needed tweaks to match the above changes.
Differential Revision: https://phabricator.services.mozilla.com/D86504
Profile buffer updates could be triggered from a number of locations, including scopes where profiler and/or system locks are held, making deadlocks possible if profiler and/or system function are called.
So instead of dispatching updates to the main thread (which may use OS task queue functions), we fold updates into a static storage. The profiler sampler loop regularly triggers processing of these pending updates.
Differential Revision: https://phabricator.services.mozilla.com/D83747
Profile buffer updates could be triggered from a number of locations, including scopes where profiler and/or system locks are held, making deadlocks possible if profiler and/or system function are called.
So instead of dispatching updates to the main thread (which may use OS task queue functions), we fold updates into a static storage. When child updates arrive, we can safely handle parent updates as well.
Child updates are assumed to arrive regularly enough to properly handle parent updates.
But in the worst case, if no updates came from children, it should mean nothing is happening, and in any case we wouldn't know how the memory is used elsewhere. Note that the chunk manager still enforces local limits automatically, so memory usage would still get limited.
Differential Revision: https://phabricator.services.mozilla.com/D83746
Knowing the time it takes for the parent process to gather its profile, we expect sub-processes not to take longer than twice that time (plus a bit more).
Each time we receive a profile, the timer is restarted* to allow more time for pending processes.
If the most current timer fires, we assume that pending processes may be frozen and unable to ever respond, so we use all profiles gathered so far.
(There is currently no indication to the user that processes are missing, this should be added in the future.)
* Tech detail: We just cancel the current timer and forget about it, then start a new one and keep track of it. When a timer fires, we can compare its address to the address of the last (most current) timer, and only that one can force the end of profile-gathering.
Differential Revision: https://phabricator.services.mozilla.com/D83626
Platform-specific observations will be able to specify a file type.
They will appear in distinct rows in the profiler.firefox.com Marker Chart.
The default type is "File", which is shown in markers as "FileIO" like before.
Differential Revision: https://phabricator.services.mozilla.com/D80399
The profiler can be "paused", which stops sampling, and since bug 1578329 stops markers as well.
Some test suites use pausing between tests (to better differentiate the tests, to keep the profiler ready to run, and to lower the amount of recorded data). But this causes problems with some tracing markers, as their matching ends have not been recorded (e.g., an end marker is missing), which show up as very loooong markers.
To solve this, we need to be able to pause sampling only, but keep recording markers.
But we still need to be able to pause the whole profiler, in particular before capturing, to avoid recording anything around that time.
This big patch is mostly mechanical changes: Wherever there are "Pause" and "Unpause/Resume" profiler functions, we add matching "PauseSampling" and "UnpauseSampling/ResumeSampling" functions that only impact the periodic sampling loop; And existing "Pause/Unpause/Resume" imply pausing sampling as well.
Exceptions and extra work:
- nsIProfiler (the JS API) already had `Pause/ResumeSampling()`, which misleadingly paused everything! Now they do the right thing, and we have `Pause/Resume()` as well.
- All tests using `Pause/ResumeSampling()` now use `Pause/Resume()`, except for Talos tests that only pause sampling between tests; Added some extra `Pause()` calls to pause everything before capturing profiles.
- GeckoJavaSampler doesn't handle pausing/resuming everything, this should be done in a follow-up bug.
- Sampling-only pauses are not streamed into JSON. If needed, we should follow-up, with potential work on the front-end to deal with these.
Differential Revision: https://phabricator.services.mozilla.com/D81492
When the tracker is destroyed, any extant channel is closed, this can trigger the rejection of pending promises, which dispatch tasks to promise handlers. So we need to ensure that this happens before threads cannot service dispatches anymore.
Differential Revision: https://phabricator.services.mozilla.com/D76436
In addition to the existing "mainthreadio" feature, we now have:
- "fileio" to also capture file I/O from other profiled threads.
- "fileioall" to also capture file I/O from all threads (even unregistered threads).
- "noiostacks" to prevent capturing stack traces for "io" markers.
These are all off by default, except for `MOZ_PROFILER_STARTUP=1`.
Differential Revision: https://phabricator.services.mozilla.com/D75764
This implement the child side:
When the first request for update arrives, it connects to the local chunk manager, to receive its updates.
If multiple updates are received, they are folded into one.
If there are both an update and a pending request, the request is fulfilled with the update and local data is reset.
And ProfilerChild handles "destroy" instructions to destroy local chunks.
At this point, the whole machinery is in place, and all combined profile buffers used in all processes should use around the maximum amount allowed.
A bit more memory may still be used, e.g., due to IPC delays, and because of recycling which keeps some unused chunks alive for later reuse. But overall that should be a small amount compared to the usual user-requested limit.
Differential Revision: https://phabricator.services.mozilla.com/D72369
The logic part of the controller receives all updates, and makes decisions to destroy old chunks when the memory limit is reached.
Differential Revision: https://phabricator.services.mozilla.com/D72368
When there is at least one ProfilerParent (i.e., we are interacting with at least one child process) AND the parent profiler is running, the ProfilerParentTracker sets up the ProfileBufferGlobalController that will manage all chunks.
As a first step, it connects with the local chunk manager (to receive chunk updates), and sends update requests to all children.
(The actual controller logic is not implemented in this patch, nor is the ProfilerChild side, see following patches.)
Differential Revision: https://phabricator.services.mozilla.com/D72367
The Gecko Profiler can notify the ProfilerParent when it's about to stop (if it was started, but the notification will happen even when it's not started).
Differential Revision: https://phabricator.services.mozilla.com/D72365
In rare cases, a dispatch may fail (e.g., threads are shutting down), we should handle this failure by rejecting the promise on the spot.
Depends on D73791
Differential Revision: https://phabricator.services.mozilla.com/D73792
It is not necessary to dispatch another task to the main thread to resolve/reject a MozPromiseHolder, because wherever this happens, the attached `Then` will do its own dispatch to the main thread.
Also, this removes a dispatch that could potentially fail, leaving the promise unfulfilled.
Depends on D73790
Differential Revision: https://phabricator.services.mozilla.com/D73791
While gathering profiles from child processes, some actions (e.g., shutting down, restarting the profiler) may reset the gathering operation.
In this case we must ensure that the promise is rejected if not already fulfilled, so that anyone waiting on it won't be blocked forever (and MozPromise enforces it in a MOZ_DIAGNOSTIC_ASSERT).
Differential Revision: https://phabricator.services.mozilla.com/D73790
This was done by:
This was done by applying:
```
diff --git a/python/mozbuild/mozbuild/code-analysis/mach_commands.py b/python/mozbuild/mozbuild/code-analysis/mach_commands.py
index 789affde7bbf..fe33c4c7d4d1 100644
--- a/python/mozbuild/mozbuild/code-analysis/mach_commands.py
+++ b/python/mozbuild/mozbuild/code-analysis/mach_commands.py
@@ -2007,7 +2007,7 @@ class StaticAnalysis(MachCommandBase):
from subprocess import Popen, PIPE, check_output, CalledProcessError
diff_process = Popen(self._get_clang_format_diff_command(commit), stdout=PIPE)
- args = [sys.executable, clang_format_diff, "-p1", "-binary=%s" % clang_format]
+ args = [sys.executable, clang_format_diff, "-p1", "-binary=%s" % clang_format, '-sort-includes']
if not output_file:
args.append("-i")
```
Then running `./mach clang-format -c <commit-hash>`
Then undoing that patch.
Then running check_spidermonkey_style.py --fixup
Then running `./mach clang-format`
I had to fix four things:
* I needed to move <utility> back down in GuardObjects.h because I was hitting
obscure problems with our system include wrappers like this:
0:03.94 /usr/include/stdlib.h:550:14: error: exception specification in declaration does not match previous declaration
0:03.94 extern void *realloc (void *__ptr, size_t __size)
0:03.94 ^
0:03.94 /home/emilio/src/moz/gecko-2/obj-debug/dist/include/malloc_decls.h:53:1: note: previous declaration is here
0:03.94 MALLOC_DECL(realloc, void*, void*, size_t)
0:03.94 ^
0:03.94 /home/emilio/src/moz/gecko-2/obj-debug/dist/include/mozilla/mozalloc.h:22:32: note: expanded from macro 'MALLOC_DECL'
0:03.94 MOZ_MEMORY_API return_type name##_impl(__VA_ARGS__);
0:03.94 ^
0:03.94 <scratch space>:178:1: note: expanded from here
0:03.94 realloc_impl
0:03.94 ^
0:03.94 /home/emilio/src/moz/gecko-2/obj-debug/dist/include/mozmemory_wrap.h:142:41: note: expanded from macro 'realloc_impl'
0:03.94 #define realloc_impl mozmem_malloc_impl(realloc)
Which I really didn't feel like digging into.
* I had to restore the order of TrustOverrideUtils.h and related files in nss
because the .inc files depend on TrustOverrideUtils.h being included earlier.
* I had to add a missing include to RollingNumber.h
* Also had to partially restore include order in JsepSessionImpl.cpp to avoid
some -WError issues due to some static inline functions being defined in a
header but not used in the rest of the compilation unit.
Differential Revision: https://phabricator.services.mozilla.com/D60327
--HG--
extra : moz-landing-system : lando
rg -l 'mozilla/Move.h' | xargs sed -i 's/#include "mozilla\/Move.h"/#include <utility>/g'
Further manual fixups and cleanups to the include order incoming.
Differential Revision: https://phabricator.services.mozilla.com/D60323
--HG--
extra : moz-landing-system : lando
Simon Giesecke
Gerald Squelart
Greg Tatum
Butkovits Atila
Florian Quèze
Kris Maglione
Csoregi Natalia
Nika Layzell
Chris Fronk
Andreas Farre
Emilio Cobos Álvarez
Jeff Walden
Nazım Can Altınova
Mihai Alexandru Michis