Consolidate Mach virtualenv management to the front of the
Mach process. This obsoletes `./mach create-mach-environment`
and simplifies the `sh` portion of the top-level `./mach` script.
This helps ensure that the Mach virtualenv doesn't become
out-of-sync and simplifies the mental model of the Mach
virtualenv situation.
Differential Revision: https://phabricator.services.mozilla.com/D120401
It was in `nativecmds` to make finding `pip3` easier.
However, since we're removing the distinction between "regular" and
"native" cmds (Mach will create and activate its venv naturally during
initialization), we need to start pruning the `nativecmds` list.
Fortunately, we can use the stored "external_python" value to decide
where to install "moz-phab" to.
The new behaviour continues to support the following use cases:
* Using Mach with system Python in the `$PATH`.
* Using Mach from within a human-managed virtualenv.
* Either of the above^ in combination with `MACH_USE_SYSTEM_PYTHON`.
Differential Revision: https://phabricator.services.mozilla.com/D120397
This version of minidump-stackwalk is now replaced with rust-minidump's
minidump-stackwalk, which we build from a FETCH. Not touching the other
stuff in this directory because I have no idea what it is.
Differential Revision: https://phabricator.services.mozilla.com/D131316
This patch adds 4 new interactive tests for facebook and reddit. The patch updates the recordings as well as needed. Furthermore, an update to the visualmetrics.py script is needed to handle a permafailure in the reddit-post tests. Note that the reddit-post test was split in 2 since they are quite large.
Differential Revision: https://phabricator.services.mozilla.com/D130665
On some platforms (e.g., Mac beta or release), native stack-walking is currently unavailable. In this case, simple tests with with no labels (or other non-native frames) would be totally empty, and would fail when expecting any data in the "samples" sections of profiles.
This is fixed by adding one label, which guarantees that stacks won't be empty.
Differential Revision: https://phabricator.services.mozilla.com/D131100
As `_run_pip()` is being removed from `VirtualenvManager` in an upcoming
patch, its usages need to be removed. Besides, they're using an
"internal" function, which is a bit of a smell.
Note that this _could_ have been solved by exposing a public `run_pip()`
function. However, I felt like that was worse because:
* Friction here is good as we try to migrate the codebase to embrace the
"requirements definition file" technique to install dependencies.
* There could be confusion about the relationship between
`install_pip_package()` (only works if venv already activated)
and `_run_pip()`, which works "in general".
Differential Revision: https://phabricator.services.mozilla.com/D130120
The expectation of 50% idle stacks was too optimistic, since we cannot control how the OS interacts with idle threads and may still had CPU time to them unexpectedly.
The patch now only expects at least one zero and at least one non-zero.
Having non-zero values should be guaranteed by the work necessary to go around the loop.
Having zero values is still hopeful, in practice we've seen a minimum of 7 out of 20 zeroes, so expecting only one should be much easier to achieve consistently. But in the worst case it could still theoretically fail; If that happens we may need to tweak again, by increasing the loop count, and/or running the whole test a few times.
Differential Revision: https://phabricator.services.mozilla.com/D131099
Suspending threads on some platforms may actually add CPU times to the target threads, artificially making it look like that thread did some work since the last CPU sampling.
This is especially visible on Linux, where suspending is done by sending a signal to the target thread, where the signal handler does quite a bit of work. On Windows there seems to be CPU work happening, possibly when the OS is checking whether a thread needs to be woken up.
To help with this, this patch effectively removes the CPU running times that happen during sampling:
* Thread sampling actions before: record-CPU stack ... record-CPU stack ...
The difference between two "record-CPU" is the amount of work done, which includes "stack" sampling.
* After: record-CPU stack reset-CPU ... record-CPU stack reset-CPU ...
Now thanks to the "reset-CPU" happening after "stack", the next "record-CPU" will only include the work done outside of the stack sampling.
This "reset-CPU" action is done in the new platform-specific function `DiscardSuspendedThreadRunningTimes`.
Note that there is a small window (just before and after the actual stack sampling) where some real CPU work may be discarded as well, but it should be negligible compared to all the work that happens between sampling loops.
Differential Revision: https://phabricator.services.mozilla.com/D130444
This is useful for the following parts, as UniqueFileHandle is a cross-platform
type which can also be used to support transferring HANDLEs between processes.
This change requires fairly sweeping changes to existing callsites, which
previously did not require owning access to the handle types when transferring.
For the most part these changes were straightforward, but manual.
Differential Revision: https://phabricator.services.mozilla.com/D126564
New features cpuallthreads, stacksallthreads, and markersallthreads now allow the user to selectively profile non-selected threads for more information.
The gtest GeckoProfiler.FeatureCombinations is modified to test combinations of up to 3 of a set of features, to lower its cost, which allows adding the new features.
Differential Revision: https://phabricator.services.mozilla.com/D130011
Now that most calls to `profiler_thread_is_being_profiled` have been updated, the `ThreadProfilingFeatures` can be made compulsory, to force callers to think about what they really want to know about the current profiling state.
Differential Revision: https://phabricator.services.mozilla.com/D130010
`profiler_thread_is_being_profiled` is used a lot for markers, so it makes sense to have a specialized version, which is a bit shorter, and lives in ProfilerMarkers.h.
Differential Revision: https://phabricator.services.mozilla.com/D130009
This replaces the simple boolean ThreadRegistrationData::mIsBeingProfiled and its directly-dependent functions.
profiler_thread_is_being_profiled now takes an extra (currently optional) argument, to check if any of the given ThreadProfilingFeatures is currently live.
This is used to control:
- Periodic sampling of CPU utilization.
- Periodic sampling of stacks.
- Markers.
This patch doesn't change the observed behavior yet (i.e., instead of IsBeingProfiled being true or false, all thread profiling features are either all or nothing), but will be used for finer-grained control in later patches.
Differential Revision: https://phabricator.services.mozilla.com/D130008
If the IOInterposer cannot retrieve a filename, the "filename" property is not even written in the marker "data" payload. So its presence should be checked before trying to use JS string functions on it.
Differential Revision: https://phabricator.services.mozilla.com/D130607
New features cpuallthreads, stacksallthreads, and markersallthreads now allow the user to selectively profile non-selected threads for more information.
The gtest GeckoProfiler.FeatureCombinations is modified to test combinations of up to 3 of a set of features, to lower its cost, which allows adding the new features.
Differential Revision: https://phabricator.services.mozilla.com/D130011
Now that most calls to `profiler_thread_is_being_profiled` have been updated, the `ThreadProfilingFeatures` can be made compulsory, to force callers to think about what they really want to know about the current profiling state.
Differential Revision: https://phabricator.services.mozilla.com/D130010
`profiler_thread_is_being_profiled` is used a lot for markers, so it makes sense to have a specialized version, which is a bit shorter, and lives in ProfilerMarkers.h.
Differential Revision: https://phabricator.services.mozilla.com/D130009
This replaces the simple boolean ThreadRegistrationData::mIsBeingProfiled and its directly-dependent functions.
profiler_thread_is_being_profiled now takes an extra (currently optional) argument, to check if any of the given ThreadProfilingFeatures is currently live.
This is used to control:
- Periodic sampling of CPU utilization.
- Periodic sampling of stacks.
- Markers.
This patch doesn't change the observed behavior yet (i.e., instead of IsBeingProfiled being true or false, all thread profiling features are either all or nothing), but will be used for finer-grained control in later patches.
Differential Revision: https://phabricator.services.mozilla.com/D130008
This is useful for the following parts, as UniqueFileHandle is a cross-platform
type which can also be used to support transferring HANDLEs between processes.
This change requires fairly sweeping changes to existing callsites, which
previously did not require owning access to the handle types when transferring.
For the most part these changes were straightforward, but manual.
Differential Revision: https://phabricator.services.mozilla.com/D126564
Removing the unlinking shouldn't be a problem, only possible side effect would be removing the partially downloaded file
Differential Revision: https://phabricator.services.mozilla.com/D130132
`GetRunningTimesWithTightTimestamp` was running the native get-running-times functions in a loop until it could run it within a "fast enough" time (based on an initial quick benchmark); This was to associate a timestamp with CPU measurements as close as possible, to get a more reliable CPU utilization graph at the end.
But this loop condition meant that if the system was under high stress, the loop could potentially re-run many times, wasting way more resources than the resulting "tightness" warrants.
So instead of repeating until "fast enough", we repeat at most once, and take the best of both measurements.
In the worst case, CPU measurements and timestamps may be a bit far apart, but it just means that a spike of activity may be reported in the next time slice instead, but the cumulative amount of work will be correct overall.
Differential Revision: https://phabricator.services.mozilla.com/D129773
Christian Holler
Mitchell Hentges
Nika Layzell
afinder
Sylvestre Ledru
Renovate Bot
Alexis Beingessner
Miko Mynttinen
Gregory Mierzwinski
Gerald Squelart
Will Kahn-Greene
Iulian Moraru
Neel Chauhan
Mark Banner
ssummar
Cristian Tuns
andrej
Myeongjun Go
Heitor Neiva
Sebastian Hengst
OnuohaOluebube
Andi-Bogdan Postelnicu
Riya
Florian Quèze