Currently the `jest/no-standalone-expect` rule needs to be disabled for `jest-in-case` cases blocks as it is not compatible.
Differential Revision: https://phabricator.services.mozilla.com/D87184
To ensure correct usage of TLSs in the profiler, they are now better encapsulated so that:
- init() is called once and its result is cached. (TLSREGISTEREDThread::Init() doesn't need proof of the PSLock, because it's using thread-safe function-static initializers.)
- get() and set() always init() as needed, or in some particular cases strongly assert that init() was successful.
Also, a null-check was missing in profiler_init_threadmanager().
Depends on D87588
Differential Revision: https://phabricator.services.mozilla.com/D87589
Assertions are also clarified with messages, to better distinguish the same tests in different locations.
Assertions should now cover all cases:
- NEW: After registering a thread in the profiler with `CorePS::AppendRegisteredThread`, the TLS should be set to that thread.
- NEW: If `profiler_register_thread` is called again, the TLS should still be set to that thread.
- When `profiler_unregister_thread` is first called, the TLS should still be set to that thread (that's the assertion currently trigering this bug 1657174),
- NEW: When `profiler_unregister_thread` is first called and after we remove the thread with `CorePS::RemoveRegisteredThread`, the TLS should now be null.
- If `profiler_unregister_thread` is called again (or with a never-registered thread), the TLS should be null.
This is a further exploratory patch for bug 1657174, this will not prevent crashes, but hopefully it should give a bit more information, at least a smaller range in which the possible presumed registration/TLS race happens.
Differential Revision: https://phabricator.services.mozilla.com/D87588
There are zero uses of this `mach` command over the past 90 days according to our telemetry. There are no external references to `mach python-safety` in-tree, and indeed if you track the history of the originating bug 1468394, it appears that once the `mach` command was created, none of the follow-up work that was discussed (i.e. running this in CI and triaging failures to appropriate owners) was done over the following 2 years.
If this ever does appear to be useful in the future, we can just resurrect this code from source control.
Differential Revision: https://phabricator.services.mozilla.com/D87351
Before this patch, we were always waiting for libxul to load because we were
starting the JVM from libxul. But we needed to start this a lot earlier. Also
thinking that JVM profiler can actually run without the gecko side, we can
start this a lot earlier than we currently start. We need to check the
environmnet variables to be able to start the profiler. It looks like the best
place to do it is inside the GeckoThread.run method.
We have also a similar code for Java debugger, with maybeWaitForJavaDebugger
name. This fucntion does similar things in terms of enviromnent variable
handling.
Differential Revision: https://phabricator.services.mozilla.com/D87069
The stack sampling can be abstracted to only use a reference to a `ProfileBuffer`, and the existing `locked_profiler_get_backtrace` can provide its stack-based `ProfileBuffer` that points at a heap-based `ProfileChunkedBuffer` (the one that will be stored in the returned `ProfilerBacktrace`).
And we can now add a public `profiler_capture_backtrace` that only takes a reference to a `ProfileChunkedBuffer`, and fills it with a backtrace.
This will be used by the new marker API, to optionally capture a backtrace in stack-based buffers at the user's request.
Differential Revision: https://phabricator.services.mozilla.com/D86514
A heap-allocate ProfileBuffer is not really needed, so it's more efficient to have one on the stack during capture, and we don't need to keep it in the `ProfilerBacktrace`.
Differential Revision: https://phabricator.services.mozilla.com/D86513
Instead of always taking ownership of both heap-allocated `ProfileBuffer` and `ProfileChunkedBuffer`, `ProfilerBacktrace` can now accept:
- Unique pointers to both or either, similar to what it was before, so a ProfilerBacktrace can be kept for later use.
- Non-owning pointers to both or either, to allow callers to use stack-based buffer(s); null pointers are allowed for totally empty backtraces.
Only the `ProfileChunkedBuffer` contains the actual data, we can create a `ProfileBuffer` on the spot if not provided.
Differential Revision: https://phabricator.services.mozilla.com/D86512
Instead of keeping a pointer to a null-terminated string, it's simpler to keep a proper `std::string`, and it helps to keep the length ready for streaming.
Differential Revision: https://phabricator.services.mozilla.com/D86511
Let `ProfilerBuffer` expose its underlying `ProfileChunkedBuffer`, this will be useful when `ProfilerBacktrace` will only be given a `ProfileBuffer`, and to perform some safety checks.
As a bonus from this change, `StoreMarker()` can be made non-generic -- It was relying on both `ProfileBuffer` and `ProfileChunkedBuffer` to have the same function `PutObjects()`. Consequently, we don't need `ProfileBuffer::PutObjects()` anymore, this removes this clunky pass-through (but useful and the best solution at the time).
Differential Revision: https://phabricator.services.mozilla.com/D86510
Until now the `ProfileBuffer` would erase its attached `ProfileChunkedBuffer` upon destruction.
However:
- The main `ProfileChunkedBuffer` is erased anyway in the `ActivePS`,
- Other `ProfileChunkedBuffer`s are short-lived and don't really need to be erased.
- The upcoming changes to `ProfilerBacktrace` and its users means that we will only keep the `ProfileChunkedBuffer` as backtrace storage, a `ProfileBuffer` will only be needed during capture and then when streaming to JSON; so we don't want the `ProfileChunkedBuffer` to be erased when detached from its capturing `ProfileBuffer`.
- Also, the erasing was done by `ResetChunkManager()` in `~ProfileBuffer()`, which was asymetric with what the constructor does (nothing!). So it's better to leave whoever did the `SetChunkManager()` to deal with the corresponding `ResetChunkManager()` (in the main case `ActivePS`, otherwise short-lived buffers being destroyed at the end of their scope).
Both `ProfileBuffer` destructors were only doing this operation, so we can just remove them completely.
Differential Revision: https://phabricator.services.mozilla.com/D86509
Backtraces and other marker data will be stored directly into a ProfileChunkedBuffer from public code in both profilers, so we will need to have the entry "kinds" available outside of the profiler directories.
This also helps with de-duplicating, since the kinds will now be in one spot and shared by both profilers.
Differential Revision: https://phabricator.services.mozilla.com/D86508
While working on this bug, I found it distracting to reason about how strings (of different char types) are stored in the profile buffer.
So instead of storing the size in bytes, I think it's better to store the string length in number of characters, in particular it matches lengths as handled by string types.
Differential Revision: https://phabricator.services.mozilla.com/D86507
The registers referenced in this patch and the assembly have been tested to
compile; the code changes have not been tested to actually work.
Differential Revision: https://phabricator.services.mozilla.com/D86612
This change doesn't resolve some of the issues in profiler code that have
x86-64-isms in them, but this is at least a start.
Differential Revision: https://phabricator.services.mozilla.com/D86595
SpliceableChunkedJSONWriter::WriteFunc was hiding base-class non-virtual JSONWriter::WriteFunc(), which made it less than ideal (for me) to reason with.
Also made a few subclasses final, to help with possible devirtualization.
Differential Revision: https://phabricator.services.mozilla.com/D86505
The main change is removing ProfileJSONWriter.cpp, making ProfileJSONWriter.h point at BaseProfileJSONWriter.h, and exposing `mozilla::baseprofiler::` classes in the top namespace as expected by users of ProfileJSONWriter.h (to minimize changes).
These two headers are now always present in the "mozilla" include directory, independent of MOZ_GECKO_PROFILER settings.
The rest is just needed tweaks to match the above changes.
Differential Revision: https://phabricator.services.mozilla.com/D86504
This makes us use one less tool from MSVC, and removes one more use of wine
in cross builds.
We replace the call to either rc/llvm-rc or windres with a wrapper script.
While the script is not strictly needed for the latter, we use a wrapper
in that case anyway because it's one step towards fixing bug 1498414.
For llvm-rc, however, we need a wrapper because llvm-rc doesn't preprocess
on its own, so the wrapper does that too.
The wrapper script also allows to deal with the default flags passed to
llvm-rc or windres, rather than inherit them from old-configure.
We also need to explicitly pass the codepage to llvm-rc, which was not
necessary with rc (presumably, llvm-rc has a different default).
While here, remove the unused WINDRES subst from js/src/old-configure.in.
Also, while here, we remove --use-temp-file, because as described in the
linked bug and in the windres manual page, it was used to work around bugs
on Windows 98 and earlier.
Differential Revision: https://phabricator.services.mozilla.com/D86312
Today we don't require that `mach` `CommandProvider`s subclass from any particular parent class and we're very lax about the requirements they must meet. While that's convenient in certain circumstances, it has some unfortunate implications for feature development.
Today the only requirements that we have for `CommandProvider`s are that they have an `__init__()` method that takes either 1 or 2 arguments, the second of which must be called `context` and is populated with the `mach` `CommandContext`. Again, while this flexibility is occasionally convenient, it is limiting. As we add features to `mach`, having a better idea what the shape of our `CommandProvider`s are and how we can instantiate them and use them is increasingly important, and this gives us additional control when having `mach` configure `CommandProvider`s based on data that is only available at the `mach` level. In particular, we plan to leverage this in bugs 985141 and 1654074.
Here we add validation to the `CommandProvider` decorator to ensure all classes inherit from `MachCommandBase`, update all `CommandProvider`s in-tree to inherit from `MachCommandBase`, and update source and test code accordingly.
Follow-up work: we now require (de facto) that the `context` be populated with a `topdir` attribute by the `populate_context_handler` function, since instantiating the `MachCommandBase` requires a `topdir` be provided. This is fine for now in the interest of keeping this patch reasonably sized, but some additional refactoring could make this cleaner.
Differential Revision: https://phabricator.services.mozilla.com/D86255
I noticed that the `objdir:build` entry in `build/virtualenv_packages.txt` entry was apparently unused. This originates from bug 841713, seven years ago, when the `objdir` handling was introduced. Today, this doesn't appear to be serving a purpose. There is no Python library in my `$objdir/build` directory; nor can I see anything in `build/moz.build` or any related files suggesting one could ever appear. I can only assume this feature has outlived its usefulness, so delete it and the relevant in-tree support.
This necessitates slightly changing the signature and implementation of the `activate_pipenv()` method; also update all callers.
Differential Revision: https://phabricator.services.mozilla.com/D85635
I noticed that the `objdir:build` entry in `build/virtualenv_packages.txt` entry was apparently unused. This originates from bug 841713, seven years ago, when the `objdir` handling was introduced. Today, this doesn't appear to be serving a purpose. There is no Python library in my `$objdir/build` directory; nor can I see anything in `build/moz.build` or any related files suggesting one could ever appear. I can only assume this feature has outlived its usefulness, so delete it and the relevant in-tree support.
This necessitates slightly changing the signature and implementation of the `activate_pipenv()` method; also update all callers.
Differential Revision: https://phabricator.services.mozilla.com/D85635
Because while the original perl script was added to add version info
to Windows binaries, it does more and will do even more with upcoming
changes.
Differential Revision: https://phabricator.services.mozilla.com/D86153
Because while the original perl script was added to add version info
to Windows binaries, it does more and will do even more with upcoming
changes.
Differential Revision: https://phabricator.services.mozilla.com/D86153
In clang-12, 5809a32e7c split `__gcov_flush` into `__gcov_dump` and `__gcov_reset`. We have to make a corresponding update in CodeCoverageHandler.cpp to avoid build errors.
Differential Revision: https://phabricator.services.mozilla.com/D86026
Add a modular approach for the integration of `static-analysis` module in order
to be able to share components of it with other modules, like the integration of
`clangd` in `vscode` where we need to have access to the configuration of `clang-tidy`
in order to have `in-ide` `static-analysis` messages.
In this initial step we make a separate module for the clang-tidy configuration.
Differential Revision: https://phabricator.services.mozilla.com/D85979
`MOZ_RELEASE_ASSERT(registeredThread == TLSRegisteredThread::RegisteredThread(lock));` is failing on Android, probably as a follow-up to bug 1651086, which allowed multiple registrations (to fix a problem with yet-unknown root cause).
This assertion is too broad to help find the issue because at this point:
- If `FindCurrentThreadRegisteredThread()` is not null, `TLSRegisteredThread::RegisteredThread()` should also not be null, and it should equal `FindCurrentThreadRegisteredThread()`, from the first succesful `profiler_register_thread()`.
- If `FindCurrentThreadRegisteredThread()` is null, `TLSRegisteredThread::RegisteredThread()` should be null as well, from the first `profiler_unregister_thread()`.
So this patch splits this assertion into multiple more targeted ones to check the above, and also some extra assertions after de-registration, to ensure that both `FindCurrentThreadRegisteredThread()` and `TLSRegisteredThread::RegisteredThread()` are null.
Differential Revision: https://phabricator.services.mozilla.com/D86118
This is not a feature-for-feature rewrite. The python version removes
unused things, and simplifies some others:
- Only two command line arguments are taken in, and all the others are
dropped and the corresponding values are gotten from the buildconfig
module instead. The command line arguments are also taken as
positional arguments rather than going with a full argument parser.
- Variable expansion in module.ver used to be limited to one specific
variable to expand for a given value, which is now replaced with the
possibility to expand any of the variables that are allowed in
module.ver.
- The perl version was adding a RT_MANIFEST entry on its own if a
manifest file existed in the objdir for the given binary, but if such
a file existed, the build would fail after linking from the changes in
bug 1613799.
- The perl version was defaulting the module name to the binary name in
a branch that was never taken because the module name was assigned to
an empty string before that.
The output from the new script has been validated to being identical to
the output from the perl script, except for one extra whitespace at the
end of a comment.
Differential Revision: https://phabricator.services.mozilla.com/D85817
This adds support for invoking the |mach logspam {report|file|bisect}|
commands. The core logic from the pypi base command is reused.
Differential Revision: https://phabricator.services.mozilla.com/D85493
This patch is the first step to our so-called "Markers 2.0" work. It's attempting to consolidate
some ad-hoc practices in the profiler into more solid practices. This is the first step to getting
the Gecko profiler to produce a format that has the timing information applied to the serialized
marker tuple, rather than a mix in the tuple and the payload.
Differential Revision: https://phabricator.services.mozilla.com/D78990
CLOSED TREE
We don't need these macros anymore, for two reasons:
1. We have static analysis to provide the same sort of checks via `MOZ_RAII`
and friends.
2. clang now warns for the "temporary that should have been a declaration" case.
The extra requirements on class construction also show up during debug tests
as performance problems.
This change was automated by using the following sed script:
```
# Remove declarations in classes.
/MOZ_DECL_USE_GUARD_OBJECT_NOTIFIER/d
/MOZ_GUARD_OBJECT_NOTIFIER_INIT/d
# Remove individual macros, carefully.
{
# We don't have to worry about substrings here because the closing
# parenthesis "anchors" the match.
s/MOZ_GUARD_OBJECT_NOTIFIER_PARAM)/)/g;
s/MOZ_GUARD_OBJECT_NOTIFIER_PARAM_TO_PARENT)/)/g;
s/MOZ_GUARD_OBJECT_NOTIFIER_PARAM_IN_IMPL)/)/g;
s/MOZ_GUARD_OBJECT_NOTIFIER_ONLY_PARAM_IN_IMPL)/)/g;
# Remove the longer identifier first.
s/MOZ_GUARD_OBJECT_NOTIFIER_ONLY_PARAM_TO_PARENT//g;
s/MOZ_GUARD_OBJECT_NOTIFIER_ONLY_PARAM//g;
}
# Remove the actual include.
\@# *include "mozilla/GuardObjects.h"@d
```
and running:
```
find . -name \*.cpp -o -name \*.h | grep -v 'GuardObjects.h' |xargs sed -i -f script 2>/dev/null
mach clang-format
```
Differential Revision: https://phabricator.services.mozilla.com/D85168
We don't need these macros anymore, for two reasons:
1. We have static analysis to provide the same sort of checks via `MOZ_RAII`
and friends.
2. clang now warns for the "temporary that should have been a declaration" case.
The extra requirements on class construction also show up during debug tests
as performance problems.
This change was automated by using the following sed script:
```
# Remove declarations in classes.
/MOZ_DECL_USE_GUARD_OBJECT_NOTIFIER/d
/MOZ_GUARD_OBJECT_NOTIFIER_INIT/d
# Remove individual macros, carefully.
{
# We don't have to worry about substrings here because the closing
# parenthesis "anchors" the match.
s/MOZ_GUARD_OBJECT_NOTIFIER_PARAM)/)/g;
s/MOZ_GUARD_OBJECT_NOTIFIER_PARAM_TO_PARENT)/)/g;
s/MOZ_GUARD_OBJECT_NOTIFIER_PARAM_IN_IMPL)/)/g;
s/MOZ_GUARD_OBJECT_NOTIFIER_ONLY_PARAM_IN_IMPL)/)/g;
# Remove the longer identifier first.
s/MOZ_GUARD_OBJECT_NOTIFIER_ONLY_PARAM_TO_PARENT//g;
s/MOZ_GUARD_OBJECT_NOTIFIER_ONLY_PARAM//g;
}
# Remove the actual include.
\@# *include "mozilla/GuardObjects.h"@d
```
and running:
```
find . -name \*.cpp -o -name \*.h | grep -v 'GuardObjects.h' |xargs sed -i -f script 2>/dev/null
mach clang-format
```
Differential Revision: https://phabricator.services.mozilla.com/D85168
These packages are all vendored in-tree, so no need to install them again.
This also fixes the issue that these packages are older than the vendored
versions, which causes issues with the use of attrs in `mozlint` now.
Differential Revision: https://phabricator.services.mozilla.com/D85037
After user ticks the checkbox of allowing error reporting, we will
report the error through the telemetry event. The event includes the
error type, XFO policy, CSP policy, the frame uri and the top-level uri.
Differential Revision: https://phabricator.services.mozilla.com/D82332
Update testing/mochitest/pywebsocket with the latest version available: pywebsocket3
is python 3 compatible. This keeps the basic structure of the old pywebsocket, but
changes the directory name to pywebsocket3 to reflect the project renaming.
Differential Revision: https://phabricator.services.mozilla.com/D84455
Start/StopAudioCallbackTracing calls are always around locked_profiler_start/stop, this patch brings these calls inside the locked_ functions.
Differential Revision: https://phabricator.services.mozilla.com/D83748
Glean generates telemetry documentation in markdown and uses tables.
This revision adds support for rendering markdown tables using an extension.
Differential Revision: https://phabricator.services.mozilla.com/D83685
Profile buffer updates could be triggered from a number of locations, including scopes where profiler and/or system locks are held, making deadlocks possible if profiler and/or system function are called.
So instead of dispatching updates to the main thread (which may use OS task queue functions), we fold updates into a static storage. The profiler sampler loop regularly triggers processing of these pending updates.
Differential Revision: https://phabricator.services.mozilla.com/D83747
Profile buffer updates could be triggered from a number of locations, including scopes where profiler and/or system locks are held, making deadlocks possible if profiler and/or system function are called.
So instead of dispatching updates to the main thread (which may use OS task queue functions), we fold updates into a static storage. When child updates arrive, we can safely handle parent updates as well.
Child updates are assumed to arrive regularly enough to properly handle parent updates.
But in the worst case, if no updates came from children, it should mean nothing is happening, and in any case we wouldn't know how the memory is used elsewhere. Note that the chunk manager still enforces local limits automatically, so memory usage would still get limited.
Differential Revision: https://phabricator.services.mozilla.com/D83746
Knowing the time it takes for the parent process to gather its profile, we expect sub-processes not to take longer than twice that time (plus a bit more).
Each time we receive a profile, the timer is restarted* to allow more time for pending processes.
If the most current timer fires, we assume that pending processes may be frozen and unable to ever respond, so we use all profiles gathered so far.
(There is currently no indication to the user that processes are missing, this should be added in the future.)
* Tech detail: We just cancel the current timer and forget about it, then start a new one and keep track of it. When a timer fires, we can compare its address to the address of the last (most current) timer, and only that one can force the end of profile-gathering.
Differential Revision: https://phabricator.services.mozilla.com/D83626
Try syntax users wishing to preserve their default selector can modify
~/.mozbuild/machrc (create it if it doesn't exist), and add:
[try]
default = syntax
Depends on D82983
Differential Revision: https://phabricator.services.mozilla.com/D82985
Since partials have started verifying signatures, the partial task has been failing in
`mach try scriptworker`. Since we are not concerned with the partial task itself,
split the tasks into two groups, so that it does not need to run.
Differential Revision: https://phabricator.services.mozilla.com/D83370
It is possible that some threads fail/forget to unregister themselves, in which case a registered thread id could get recycled by a later thread, which was not allowed before this patch.
Note: The thread name cannot currently be changed. We record a special marker with the new name, so the frontend could process it to split the thread track at that point.
We also record a marker when profiler_unregister_thread is called from an already-unregistered thread, this could help find reg/unreg mismatches or nesting in Firefox threads.
Differential Revision: https://phabricator.services.mozilla.com/D83293
On Linux (including Android), it was assumed that a registered thread could always be suspended through `tgkill`.
However in some cases a thread may not be correctly unregistered, in which case this would trigger `MOZ_ASSERT` or wait forever in the following loop.
This will especially be needed when `profiler_{,un}register_thread()` are made less strict in the following patch.
Windows and Mac already handle suspension failures.
Differential Revision: https://phabricator.services.mozilla.com/D83292
Add an action that will trigger a task that runs
`mach release push-scriptworker-canary`
to test a new scriptworker deployment.
Differential Revision: https://phabricator.services.mozilla.com/D82821
Adds some additional assertions against what would be ran with |mach try auto|.
This also fixes a legit bug that the 'test_important_manifests_only' test
caught.
Differential Revision: https://phabricator.services.mozilla.com/D81404
Emil Farisan
Duncan Dean
Bogdan Tara
Gerald Squelart
Simon Giesecke
Florian Quèze
Sonia Singla
Ricky Stewart
Nazım Can Altınova
Andrew Halberstadt
Sylvestre Ledru
Nathan Froyd
Michael Hoffmann
emcminn
Mike Hommey
Andreea Pavel
Evan Welsh
Hamzah
Mihai Alexandru Michis
David Major
Andi-Bogdan Postelnicu
championshuttler
Tom Prince
Eric Rahm
Julian Descottes
neil
Christian Holler
Ben Hearsum
Tyson Smith
Myeongjun Go
Mark Banner
Narcis Beleuzu
Jonathan Almeida [:jonalmeida]
Tim Huang
Geoff Brown
Jonathan Kew
Mitchell Hentges
Tomislav Jovanovic
Randell Jesup
Csoregi Natalia
Andrew Halberstadt
Dan Mosedale
Harry Twyford
Cosmin Sabou
Kris Maglione
Axel Hecht