This patch adds 4 new interactive tests for facebook and reddit. The patch updates the recordings as well as needed. Furthermore, an update to the visualmetrics.py script is needed to handle a permafailure in the reddit-post tests. Note that the reddit-post test was split in 2 since they are quite large.
Differential Revision: https://phabricator.services.mozilla.com/D130665
On some platforms (e.g., Mac beta or release), native stack-walking is currently unavailable. In this case, simple tests with with no labels (or other non-native frames) would be totally empty, and would fail when expecting any data in the "samples" sections of profiles.
This is fixed by adding one label, which guarantees that stacks won't be empty.
Differential Revision: https://phabricator.services.mozilla.com/D131100
As `_run_pip()` is being removed from `VirtualenvManager` in an upcoming
patch, its usages need to be removed. Besides, they're using an
"internal" function, which is a bit of a smell.
Note that this _could_ have been solved by exposing a public `run_pip()`
function. However, I felt like that was worse because:
* Friction here is good as we try to migrate the codebase to embrace the
"requirements definition file" technique to install dependencies.
* There could be confusion about the relationship between
`install_pip_package()` (only works if venv already activated)
and `_run_pip()`, which works "in general".
Differential Revision: https://phabricator.services.mozilla.com/D130120
The expectation of 50% idle stacks was too optimistic, since we cannot control how the OS interacts with idle threads and may still had CPU time to them unexpectedly.
The patch now only expects at least one zero and at least one non-zero.
Having non-zero values should be guaranteed by the work necessary to go around the loop.
Having zero values is still hopeful, in practice we've seen a minimum of 7 out of 20 zeroes, so expecting only one should be much easier to achieve consistently. But in the worst case it could still theoretically fail; If that happens we may need to tweak again, by increasing the loop count, and/or running the whole test a few times.
Differential Revision: https://phabricator.services.mozilla.com/D131099
Suspending threads on some platforms may actually add CPU times to the target threads, artificially making it look like that thread did some work since the last CPU sampling.
This is especially visible on Linux, where suspending is done by sending a signal to the target thread, where the signal handler does quite a bit of work. On Windows there seems to be CPU work happening, possibly when the OS is checking whether a thread needs to be woken up.
To help with this, this patch effectively removes the CPU running times that happen during sampling:
* Thread sampling actions before: record-CPU stack ... record-CPU stack ...
The difference between two "record-CPU" is the amount of work done, which includes "stack" sampling.
* After: record-CPU stack reset-CPU ... record-CPU stack reset-CPU ...
Now thanks to the "reset-CPU" happening after "stack", the next "record-CPU" will only include the work done outside of the stack sampling.
This "reset-CPU" action is done in the new platform-specific function `DiscardSuspendedThreadRunningTimes`.
Note that there is a small window (just before and after the actual stack sampling) where some real CPU work may be discarded as well, but it should be negligible compared to all the work that happens between sampling loops.
Differential Revision: https://phabricator.services.mozilla.com/D130444
This is useful for the following parts, as UniqueFileHandle is a cross-platform
type which can also be used to support transferring HANDLEs between processes.
This change requires fairly sweeping changes to existing callsites, which
previously did not require owning access to the handle types when transferring.
For the most part these changes were straightforward, but manual.
Differential Revision: https://phabricator.services.mozilla.com/D126564
New features cpuallthreads, stacksallthreads, and markersallthreads now allow the user to selectively profile non-selected threads for more information.
The gtest GeckoProfiler.FeatureCombinations is modified to test combinations of up to 3 of a set of features, to lower its cost, which allows adding the new features.
Differential Revision: https://phabricator.services.mozilla.com/D130011
Now that most calls to `profiler_thread_is_being_profiled` have been updated, the `ThreadProfilingFeatures` can be made compulsory, to force callers to think about what they really want to know about the current profiling state.
Differential Revision: https://phabricator.services.mozilla.com/D130010
`profiler_thread_is_being_profiled` is used a lot for markers, so it makes sense to have a specialized version, which is a bit shorter, and lives in ProfilerMarkers.h.
Differential Revision: https://phabricator.services.mozilla.com/D130009
This replaces the simple boolean ThreadRegistrationData::mIsBeingProfiled and its directly-dependent functions.
profiler_thread_is_being_profiled now takes an extra (currently optional) argument, to check if any of the given ThreadProfilingFeatures is currently live.
This is used to control:
- Periodic sampling of CPU utilization.
- Periodic sampling of stacks.
- Markers.
This patch doesn't change the observed behavior yet (i.e., instead of IsBeingProfiled being true or false, all thread profiling features are either all or nothing), but will be used for finer-grained control in later patches.
Differential Revision: https://phabricator.services.mozilla.com/D130008
If the IOInterposer cannot retrieve a filename, the "filename" property is not even written in the marker "data" payload. So its presence should be checked before trying to use JS string functions on it.
Differential Revision: https://phabricator.services.mozilla.com/D130607
New features cpuallthreads, stacksallthreads, and markersallthreads now allow the user to selectively profile non-selected threads for more information.
The gtest GeckoProfiler.FeatureCombinations is modified to test combinations of up to 3 of a set of features, to lower its cost, which allows adding the new features.
Differential Revision: https://phabricator.services.mozilla.com/D130011
Now that most calls to `profiler_thread_is_being_profiled` have been updated, the `ThreadProfilingFeatures` can be made compulsory, to force callers to think about what they really want to know about the current profiling state.
Differential Revision: https://phabricator.services.mozilla.com/D130010
`profiler_thread_is_being_profiled` is used a lot for markers, so it makes sense to have a specialized version, which is a bit shorter, and lives in ProfilerMarkers.h.
Differential Revision: https://phabricator.services.mozilla.com/D130009
This replaces the simple boolean ThreadRegistrationData::mIsBeingProfiled and its directly-dependent functions.
profiler_thread_is_being_profiled now takes an extra (currently optional) argument, to check if any of the given ThreadProfilingFeatures is currently live.
This is used to control:
- Periodic sampling of CPU utilization.
- Periodic sampling of stacks.
- Markers.
This patch doesn't change the observed behavior yet (i.e., instead of IsBeingProfiled being true or false, all thread profiling features are either all or nothing), but will be used for finer-grained control in later patches.
Differential Revision: https://phabricator.services.mozilla.com/D130008
This is useful for the following parts, as UniqueFileHandle is a cross-platform
type which can also be used to support transferring HANDLEs between processes.
This change requires fairly sweeping changes to existing callsites, which
previously did not require owning access to the handle types when transferring.
For the most part these changes were straightforward, but manual.
Differential Revision: https://phabricator.services.mozilla.com/D126564
Removing the unlinking shouldn't be a problem, only possible side effect would be removing the partially downloaded file
Differential Revision: https://phabricator.services.mozilla.com/D130132
`GetRunningTimesWithTightTimestamp` was running the native get-running-times functions in a loop until it could run it within a "fast enough" time (based on an initial quick benchmark); This was to associate a timestamp with CPU measurements as close as possible, to get a more reliable CPU utilization graph at the end.
But this loop condition meant that if the system was under high stress, the loop could potentially re-run many times, wasting way more resources than the resulting "tightness" warrants.
So instead of repeating until "fast enough", we repeat at most once, and take the best of both measurements.
In the worst case, CPU measurements and timestamps may be a bit far apart, but it just means that a spike of activity may be reported in the next time slice instead, but the cumulative amount of work will be correct overall.
Differential Revision: https://phabricator.services.mozilla.com/D129773
Changed wget output log granularity from 📣 to :giga: (only used in case the python library didn't cover all usecases)
Added python methods to async cache/download all required files for update verify
Modified Dockerfile requirements to include aiohttp python lib
Differential Revision: https://phabricator.services.mozilla.com/D129429
Sylvestre Ledru
Miko Mynttinen
Gregory Mierzwinski
Gerald Squelart
Mitchell Hentges
Will Kahn-Greene
Renovate Bot
Iulian Moraru
Neel Chauhan
Nika Layzell
Mark Banner
ssummar
Cristian Tuns
andrej
Myeongjun Go
Heitor Neiva
Sebastian Hengst
OnuohaOluebube
Andi-Bogdan Postelnicu
Riya
Florian Quèze
Tooru Fujisawa
Botond Ballo
Jan Odvarko
Evgenia Kotovich
criss