This patch is the first step to our so-called "Markers 2.0" work. It's attempting to consolidate
some ad-hoc practices in the profiler into more solid practices. This is the first step to getting
the Gecko profiler to produce a format that has the timing information applied to the serialized
marker tuple, rather than a mix in the tuple and the payload.
Differential Revision: https://phabricator.services.mozilla.com/D78990
CLOSED TREE
We don't need these macros anymore, for two reasons:
1. We have static analysis to provide the same sort of checks via `MOZ_RAII`
and friends.
2. clang now warns for the "temporary that should have been a declaration" case.
The extra requirements on class construction also show up during debug tests
as performance problems.
This change was automated by using the following sed script:
```
# Remove declarations in classes.
/MOZ_DECL_USE_GUARD_OBJECT_NOTIFIER/d
/MOZ_GUARD_OBJECT_NOTIFIER_INIT/d
# Remove individual macros, carefully.
{
# We don't have to worry about substrings here because the closing
# parenthesis "anchors" the match.
s/MOZ_GUARD_OBJECT_NOTIFIER_PARAM)/)/g;
s/MOZ_GUARD_OBJECT_NOTIFIER_PARAM_TO_PARENT)/)/g;
s/MOZ_GUARD_OBJECT_NOTIFIER_PARAM_IN_IMPL)/)/g;
s/MOZ_GUARD_OBJECT_NOTIFIER_ONLY_PARAM_IN_IMPL)/)/g;
# Remove the longer identifier first.
s/MOZ_GUARD_OBJECT_NOTIFIER_ONLY_PARAM_TO_PARENT//g;
s/MOZ_GUARD_OBJECT_NOTIFIER_ONLY_PARAM//g;
}
# Remove the actual include.
\@# *include "mozilla/GuardObjects.h"@d
```
and running:
```
find . -name \*.cpp -o -name \*.h | grep -v 'GuardObjects.h' |xargs sed -i -f script 2>/dev/null
mach clang-format
```
Differential Revision: https://phabricator.services.mozilla.com/D85168
We don't need these macros anymore, for two reasons:
1. We have static analysis to provide the same sort of checks via `MOZ_RAII`
and friends.
2. clang now warns for the "temporary that should have been a declaration" case.
The extra requirements on class construction also show up during debug tests
as performance problems.
This change was automated by using the following sed script:
```
# Remove declarations in classes.
/MOZ_DECL_USE_GUARD_OBJECT_NOTIFIER/d
/MOZ_GUARD_OBJECT_NOTIFIER_INIT/d
# Remove individual macros, carefully.
{
# We don't have to worry about substrings here because the closing
# parenthesis "anchors" the match.
s/MOZ_GUARD_OBJECT_NOTIFIER_PARAM)/)/g;
s/MOZ_GUARD_OBJECT_NOTIFIER_PARAM_TO_PARENT)/)/g;
s/MOZ_GUARD_OBJECT_NOTIFIER_PARAM_IN_IMPL)/)/g;
s/MOZ_GUARD_OBJECT_NOTIFIER_ONLY_PARAM_IN_IMPL)/)/g;
# Remove the longer identifier first.
s/MOZ_GUARD_OBJECT_NOTIFIER_ONLY_PARAM_TO_PARENT//g;
s/MOZ_GUARD_OBJECT_NOTIFIER_ONLY_PARAM//g;
}
# Remove the actual include.
\@# *include "mozilla/GuardObjects.h"@d
```
and running:
```
find . -name \*.cpp -o -name \*.h | grep -v 'GuardObjects.h' |xargs sed -i -f script 2>/dev/null
mach clang-format
```
Differential Revision: https://phabricator.services.mozilla.com/D85168
Start/StopAudioCallbackTracing calls are always around locked_profiler_start/stop, this patch brings these calls inside the locked_ functions.
Differential Revision: https://phabricator.services.mozilla.com/D83748
Profile buffer updates could be triggered from a number of locations, including scopes where profiler and/or system locks are held, making deadlocks possible if profiler and/or system function are called.
So instead of dispatching updates to the main thread (which may use OS task queue functions), we fold updates into a static storage. The profiler sampler loop regularly triggers processing of these pending updates.
Differential Revision: https://phabricator.services.mozilla.com/D83747
Profile buffer updates could be triggered from a number of locations, including scopes where profiler and/or system locks are held, making deadlocks possible if profiler and/or system function are called.
So instead of dispatching updates to the main thread (which may use OS task queue functions), we fold updates into a static storage. When child updates arrive, we can safely handle parent updates as well.
Child updates are assumed to arrive regularly enough to properly handle parent updates.
But in the worst case, if no updates came from children, it should mean nothing is happening, and in any case we wouldn't know how the memory is used elsewhere. Note that the chunk manager still enforces local limits automatically, so memory usage would still get limited.
Differential Revision: https://phabricator.services.mozilla.com/D83746
Knowing the time it takes for the parent process to gather its profile, we expect sub-processes not to take longer than twice that time (plus a bit more).
Each time we receive a profile, the timer is restarted* to allow more time for pending processes.
If the most current timer fires, we assume that pending processes may be frozen and unable to ever respond, so we use all profiles gathered so far.
(There is currently no indication to the user that processes are missing, this should be added in the future.)
* Tech detail: We just cancel the current timer and forget about it, then start a new one and keep track of it. When a timer fires, we can compare its address to the address of the last (most current) timer, and only that one can force the end of profile-gathering.
Differential Revision: https://phabricator.services.mozilla.com/D83626
It is possible that some threads fail/forget to unregister themselves, in which case a registered thread id could get recycled by a later thread, which was not allowed before this patch.
Note: The thread name cannot currently be changed. We record a special marker with the new name, so the frontend could process it to split the thread track at that point.
We also record a marker when profiler_unregister_thread is called from an already-unregistered thread, this could help find reg/unreg mismatches or nesting in Firefox threads.
Differential Revision: https://phabricator.services.mozilla.com/D83293
On Linux (including Android), it was assumed that a registered thread could always be suspended through `tgkill`.
However in some cases a thread may not be correctly unregistered, in which case this would trigger `MOZ_ASSERT` or wait forever in the following loop.
This will especially be needed when `profiler_{,un}register_thread()` are made less strict in the following patch.
Windows and Mac already handle suspension failures.
Differential Revision: https://phabricator.services.mozilla.com/D83292
Platform-specific observations will be able to specify a file type.
They will appear in distinct rows in the profiler.firefox.com Marker Chart.
The default type is "File", which is shown in markers as "FileIO" like before.
Differential Revision: https://phabricator.services.mozilla.com/D80399
The profiler can be "paused", which stops sampling, and since bug 1578329 stops markers as well.
Some test suites use pausing between tests (to better differentiate the tests, to keep the profiler ready to run, and to lower the amount of recorded data). But this causes problems with some tracing markers, as their matching ends have not been recorded (e.g., an end marker is missing), which show up as very loooong markers.
To solve this, we need to be able to pause sampling only, but keep recording markers.
But we still need to be able to pause the whole profiler, in particular before capturing, to avoid recording anything around that time.
This big patch is mostly mechanical changes: Wherever there are "Pause" and "Unpause/Resume" profiler functions, we add matching "PauseSampling" and "UnpauseSampling/ResumeSampling" functions that only impact the periodic sampling loop; And existing "Pause/Unpause/Resume" imply pausing sampling as well.
Exceptions and extra work:
- nsIProfiler (the JS API) already had `Pause/ResumeSampling()`, which misleadingly paused everything! Now they do the right thing, and we have `Pause/Resume()` as well.
- All tests using `Pause/ResumeSampling()` now use `Pause/Resume()`, except for Talos tests that only pause sampling between tests; Added some extra `Pause()` calls to pause everything before capturing profiles.
- GeckoJavaSampler doesn't handle pausing/resuming everything, this should be done in a follow-up bug.
- Sampling-only pauses are not streamed into JSON. If needed, we should follow-up, with potential work on the front-end to deal with these.
Differential Revision: https://phabricator.services.mozilla.com/D81492
This adds 3 new profiler markers for each IPC message:
* One just before the first byte is sent over the IPC channel
* One just after the last byte is sent over the IPC channel
* One just after the last byte is received from the IPC channel
With the already-existing IPC markers (for when SendXXX and RecvXXX are
called), this allows us to calculate the following statistics:
* Send thread latency
* IPC send duration
* IPC recv latency
* Recv thread latency
For more information on how this is presented in the UI, see:
<https://github.com/firefox-devtools/profiler/pull/2535>.
Differential Revision: https://phabricator.services.mozilla.com/D70790
Some profile meta information can be gathered before the profiler mutex must be locked.
This reduces the main-thread locking when capturing.
Most importantly, it prevents deadlocking in case any of the data-gathering operation would itself rely on profiler-locking functions (e.g., starting a thread, which requires the lock to register the new thread with the profiler).
Differential Revision: https://phabricator.services.mozilla.com/D81491
Some profile meta information can be gathered before the profiler mutex must be locked.
This reduces the main-thread locking when capturing.
Most importantly, it prevents deadlocking in case any of the data-gathering operation would itself rely on profiler-locking functions (e.g., starting a thread, which requires the lock to register the new thread with the profiler).
Differential Revision: https://phabricator.services.mozilla.com/D81491
This was previously r=jchen and landed, but was backed out because some
Android tests were running on an unrealistically old Linux kernel. These
tests have been retired, so this can reland.
Differential Revision: https://phabricator.services.mozilla.com/D81278
Bug 1486524 disabled the features from the custom linker that required
AutoObjectMapperFaultyLib. We can now rely on AutoObjectMapperPOSIX
instead.
Differential Revision: https://phabricator.services.mozilla.com/D81264
The Base Profiler is still recent and barely used, so it may contain some bugs.
With bug 1586939, the Base Profiler is now used more often because it is controlled the same way as the Gecko Profiler.
This has surfaced some intermittent issues, which pollute existing tests.
Until the root cause is found (see bug 1648325), setting `MOZ_PROFILER_STARTUP_NO_BASE=1` prevents the Base Profiler from running. This may be used where problems are visible, to diagnostic them and/or reduce them where needed.
Differential Revision: https://phabricator.services.mozilla.com/D81018
To avoid deadlocks between the profiler mutex and another mutex that may be locked when an allocation is intercepted, we now abort this interception if the profiler mutex is already locked.
Differential Revision: https://phabricator.services.mozilla.com/D79413
This function can be called whether the profiler mutex is locked or not. If locked, the provided pointer to the profiler mutex is used internally, otherwise the mutex will be locked as needed.
Differential Revision: https://phabricator.services.mozilla.com/D79412
PSAutoTryLock attempts to lock the mutex, but gives up immediately if it's already locked.
This can be used in profiler functions that may be called from unknown code, to avoid deadlocks involving the profiler mutex and another mutex that could be indirectly used by the profiler itself.
Differential Revision: https://phabricator.services.mozilla.com/D79410