This was previously r=jchen and landed, but was backed out because some
Android tests were running on an unrealistically old Linux kernel. These
tests have been retired, so this can reland.
Differential Revision: https://phabricator.services.mozilla.com/D81278
Bug 1486524 disabled the features from the custom linker that required
AutoObjectMapperFaultyLib. We can now rely on AutoObjectMapperPOSIX
instead.
Differential Revision: https://phabricator.services.mozilla.com/D81264
The Base Profiler is still recent and barely used, so it may contain some bugs.
With bug 1586939, the Base Profiler is now used more often because it is controlled the same way as the Gecko Profiler.
This has surfaced some intermittent issues, which pollute existing tests.
Until the root cause is found (see bug 1648325), setting `MOZ_PROFILER_STARTUP_NO_BASE=1` prevents the Base Profiler from running. This may be used where problems are visible, to diagnostic them and/or reduce them where needed.
Differential Revision: https://phabricator.services.mozilla.com/D81018
To avoid deadlocks between the profiler mutex and another mutex that may be locked when an allocation is intercepted, we now abort this interception if the profiler mutex is already locked.
Differential Revision: https://phabricator.services.mozilla.com/D79413
This function can be called whether the profiler mutex is locked or not. If locked, the provided pointer to the profiler mutex is used internally, otherwise the mutex will be locked as needed.
Differential Revision: https://phabricator.services.mozilla.com/D79412
PSAutoTryLock attempts to lock the mutex, but gives up immediately if it's already locked.
This can be used in profiler functions that may be called from unknown code, to avoid deadlocks involving the profiler mutex and another mutex that could be indirectly used by the profiler itself.
Differential Revision: https://phabricator.services.mozilla.com/D79410
There is some remaining code in central originating from bug 1184405, which sought to associate source files with their "affected" test files. That ended up not panning out, and bug 1644228 removed a lot of that code, but left some remnants in the `Files` object which are still referenced in a couple different places. I'm deleting all of that code in `context.py` plus everything that references it for the following reasons:
1. Right now, `Files.{test_files,test_tags,test_flavors}` do get populated, but only ever with "default" values -- namely `moz.build` files that are above the files in question in the directory hierarchy. This is a heuristic that doesn't actually have anything to do with mapping source files to their corresponding test files, which is misleading.
2. Those attributes are accessed in two places. The first is in the `mach file-info dep-tests` command. This command isn't referenced anywhere else in tree and I don't have any evidence anyone ever uses it. Even if they do, I would claim that doing so is a mistake (because the results of the command aren't meaningful and are just populated by the "defaults" described above), and that person's workflow should be migrated to something else that *is* meaningful.
3. The second place where this metadata is accessed is in `testing/mozbase/moztest/moztest/resolve.py`; that method is invoked in `tools/tryselect/selectors/syntax.py`, but only if you pass `--detect-paths` to `mach try syntax`. This is [entirely broken](https://bugzilla.mozilla.org/show_bug.cgi?id=1614614), and even if we made an effort to fix it, it wouldn't do anything resembling what the documentation of `--detect-paths` suggests it would do (again, because the data isn't populated meaningfully). So I'm deleting the command line option entirely.
Differential Revision: https://phabricator.services.mozilla.com/D79711
The version check is done above. So, it will fail there.
Move the binary check earlier to have a better error message
+ simplify the code
Differential Revision: https://phabricator.services.mozilla.com/D79725
Note that I cannot do unit test because of it doesn't support regex payload
''Failed: could not resolve a lint function from''
Differential Revision: https://phabricator.services.mozilla.com/D78158
In the JIT frame sampler, we apply the appropriate category in addition to
the "implementation" field. For JS frames (IS_JS_FRAME), we identify as
either BaselineInterpreter or Interpreter. Note that JS_Other still applies
to various places we enter SpiderMonkey outside of RunScript.
Differential Revision: https://phabricator.services.mozilla.com/D79524
Replace the duplicate lists in mozglue/baseprofiler/public and js/public with
a shared list. Add this list to both moz.build files so it is published twice
which simplifies supporting different standalone configurations.
Differential Revision: https://phabricator.services.mozilla.com/D79520
Introduce an IS_BLINTERP_FRAME flag to ProfilingStackFrame to distinguish C++
and Baseline interpreter frames. In the profile data this sets the
"implementation" to "blinterp".
Differential Revision: https://phabricator.services.mozilla.com/D78725
Now we are pausing the profiler just before capturing the profile data. Android
sampler will never be running when this code is being run
Depends on D78037
Differential Revision: https://phabricator.services.mozilla.com/D78038
This patch adds a small README file to the tools/browsertime folder to give people more information about its purpose and provides a link to the perftest wiki.
Depends on D78044
Differential Revision: https://phabricator.services.mozilla.com/D78045
When python2 interprets headers here, it will attempt to encode values into a string in a blanket fashion.
In this case, the `Accept` header is encoded into an invalid value, but it "works".
Python3 validates more strictly, and throws an error when an array is encountered as a value here. Formatting it
as a valid string instead resolves the problem.
Differential Revision: https://phabricator.services.mozilla.com/D77735
With the addition of this change, a lone `mach busted file` will throw an error (since this should only ever be used by actual humans, and not in automation or anything, the backwards-compatibility breakage isn't a huge deal). Now it's expected to pass in `mach busted file $COMMAND`, where $COMMAND is the mach command that you were running when the error occurred, and we'll figure out which bugzilla component to file the bug against for you instead of directing it to `Firefox Build System :: General` regardless of whether the issue has *anything* to do with the build system. We preserve `mach busted file general` as a backup command that doesn't do any of this heavy logic.
Differential Revision: https://phabricator.services.mozilla.com/D77538
`|mach lint -f treeherder|` expects lint to not fail ever, if a lint fails it's
just ignored. This means that when our javadoc errors out and fails the build
all lints are ignored.
To avoid that we add `failOnError = false` to the javadoc job so that we can
finish the build and actually report the errors.
This patch also adds gradle output `|mach lint|` to make it easier to
diagnose errors in the lint.
Differential Revision: https://phabricator.services.mozilla.com/D77373
`|mach lint -f treeherder|` expects lints to not fail ever, if a lint fails it's
just ignored. This means that when our javadoc errors out and fails the build
all lints are ignored.
To avoid that we add `failOnError = false` to the javadoc job so that we can
finish the build and actually report the errors.
This patch also adds gradle output `|mach lint|` to make it easier to
diagnose errors in the lint.
Differential Revision: https://phabricator.services.mozilla.com/D77373
nsTHashtable::Remove doesn't assert if the item isn't present. Match that
behavior by removing the assert and putting it at all of the call sites.
This just turns IDMap::Remove into RemoveIfPresent, so merge them.
Differential Revision: https://phabricator.services.mozilla.com/D77166
This function is similar to the Put() method in nsTHashtable, but it lists the
key second and it asserts that the key is not already in the map. This patch
swaps the arguments and hoists the assertion out, where appropriate. Note that
there are a few places that were working around this assert, so for those places
don't include the assert.
Differential Revision: https://phabricator.services.mozilla.com/D77165
Now both profilers are controlled by the same environment variables, e.g.
"MOZ_PROFILER_STARTUP" to run both profilers at startup.
The only remaining Base Profiler-specific commands are:
- MOZ_BASE_PROFILER_HELP, because the Base Profiler doesn't have the same
capabilities as Gecko, so it may display different available features.
- MOZ_BASE_PROFILER_LOGGING, because the Base Profiler doesn't have access to
MOZ_LOG (both the env-var and the C++ macros).
Differential Revision: https://phabricator.services.mozilla.com/D54449
Now both profilers are controlled by the same environment variables, e.g.
"MOZ_PROFILER_STARTUP" to run both profilers at startup.
The only remaining Base Profiler-specific commands are:
- MOZ_BASE_PROFILER_HELP, because the Base Profiler doesn't have the same
capabilities as Gecko, so it may display different available features.
- MOZ_BASE_PROFILER_LOGGING, because the Base Profiler doesn't have access to
MOZ_LOG (both the env-var and the C++ macros).
Differential Revision: https://phabricator.services.mozilla.com/D54449
Hopefully this will avoid some intermittent failures that occur when
`fix-stacks` is deleted at the end of a test run.
Differential Revision: https://phabricator.services.mozilla.com/D76980
If EAF+ is enabled for firefox.exe, the process does not launch because we parse
the PE headers of ntdll.dll at startup, which is prohibited by EAF+.
With this patch, we skip two operations when EAF+ is enabled.
The first one is to cache ntdll's IAT at startup. Because EAF+ is expected to
prevent an injected module from parsing PE headers and modifying IAT, we can skip
this caching safely.
The second one is to load ntdll's debug information for the profiler. With this
patch, the profiler's callstack will not show a raw address instead of a symbol
name. It's a bad side effect, but much better than startup crash.
Differential Revision: https://phabricator.services.mozilla.com/D76959
This is the logic of tracing the WebGPU API calls at the level of wgpu-core,
serialized into a folder of choosing on the user drive. Traces are extremely portable,
they can be shared (on BugZilla) and then replayed on the developer machine,
which can have a different architecture from the users machine.
The standalone player is introduced in `gfx/wgpu/player`, similar to WebRender's Wrench.
The output dir is controlled by "dom.webgpu.traceDir" pref. No tracing happens if it's empty.
Differential Revision: https://phabricator.services.mozilla.com/D73333
This is the logic of tracing the WebGPU API calls at the level of wgpu-core,
serialized into a folder of choosing on the user drive. Traces are extremely portable,
they can be shared (on BugZilla) and then replayed on the developer machine,
which can have a different architecture from the users machine.
The standalone player is introduced in `gfx/wgpu/player`, similar to WebRender's Wrench.
The output dir is controlled by "dom.webgpu.traceDir" pref. No tracing happens if it's empty.
Differential Revision: https://phabricator.services.mozilla.com/D73333
This code was not being built any more.
GeckoView requires Android OS version 4.3 (API Level 18). I'm not sure what the
corresponding ANDROID_VERSION macro constant for API Level 18 is, but it's
definitely higher than 16.
Differential Revision: https://phabricator.services.mozilla.com/D73940
Also change the processor IDs to hex, because that's what the Linux kernel
source code uses. And reorder the cases a little.
Differential Revision: https://phabricator.services.mozilla.com/D76272
When the tracker is destroyed, any extant channel is closed, this can trigger the rejection of pending promises, which dispatch tasks to promise handlers. So we need to ensure that this happens before threads cannot service dispatches anymore.
Differential Revision: https://phabricator.services.mozilla.com/D76436
In addition to the existing "mainthreadio" feature, we now have:
- "fileio" to also capture file I/O from other profiled threads.
- "fileioall" to also capture file I/O from all threads (even unregistered threads).
- "noiostacks" to prevent capturing stack traces for "io" markers.
These are all off by default, except for `MOZ_PROFILER_STARTUP=1`.
Differential Revision: https://phabricator.services.mozilla.com/D75764
This can be used while the profiler is running, to know if the current thread is registered -- regardless of whether it is actively profiled.
This will help distinguish registered but non-profiled threads from threads that are not even registered (e.g., OS-generated threads).
Differential Revision: https://phabricator.services.mozilla.com/D76281
Some functions may want to examine more than one feature at once, these function return everything (if the profiler is active, and optionally unpaused) so that only one call is needed.
Differential Revision: https://phabricator.services.mozilla.com/D76280
This is a simple compile-time check that the feature numbers go from 0 to N-1, to prevent some mistakes when updating feature lists.
Differential Revision: https://phabricator.services.mozilla.com/D75763
The FileIO marker may be recorded into the main thread, so we need to keep track of the thread in which the IO really happened (which may be a thread that's not even selected for profiling).
There is not JSON output for it yet.
Differential Revision: https://phabricator.services.mozilla.com/D75761
Some markers may be more useful when gathered into a single track, and the main thread is ideal because it's almost always present.
Differential Revision: https://phabricator.services.mozilla.com/D75760
I need to add symbolication support for the mochitest Gecko Profiler command line
option. These profiles also need to be symbolicated. Unfortunately, there is not
a common place where I could use these files. Talos and Raptor each had their
own copy of the snappy symbolication server.
This commit consolidates these packages into a re-usable mozbase package that can
be used in mochitests, and eventually in other places like xpcshell tests.
I stubbed out a test file, but it doesn't do anything quite yet. This commit makes
it so that the tests still work in Raptor and Talos, but doesn't add any features.
It also doesn't try too hard to make the files look like a mozbase package.
Differential Revision: https://phabricator.services.mozilla.com/D74289
Instead of accepting a `UniquePtr<ProfilerMarkerPayload>`, `profiler_add_marker_for_thread` now takes it by `const ProfilerMarkerPayload&`.
All callers can now create the payload on the stack, or as a temporary object. This saves a memory alloc+free for each call.
Differential Revision: https://phabricator.services.mozilla.com/D75912
This patch will do :
- add `Media` markers in related codes
- implement `MediaSampleMarkerPayload` to record the media sample
The advantage of doing so :
- using markers can help us know what happens on the media by a glance without expanding the call stack
- adding sample markers allows us compare the speed of decoding sample in `MediaDecoderStataMachine` and rendering sample in `VideoSink`
Differential Revision: https://phabricator.services.mozilla.com/D74174
In this bug we're moving away from monolithic JNI headers to class-specific
headers so that we don't have to rebuild the world every time we make a change
to a JNI interface.
Differential Revision: https://phabricator.services.mozilla.com/D75368
This patch is adding an option to push a perftest run in the CI.
It's based on :
- sparse profiles
- push_to_try
- options passed through try_task_config.json
Differential Revision: https://phabricator.services.mozilla.com/D74115
Mach doesn't know which tasks are part of a release - that's decided in the decision task. Instead of
showing a bogus estimation, we shouldn't show one at all.
Differential Revision: https://phabricator.services.mozilla.com/D74904
In some NT-specific code, a list of "items()" was being updated. In python 3, to modify the result
of "items()", you have to gain ownership first by explicitly converting it into a list().
Also resolves some flaky failures that were seen locally by marking test_presets.py as sequential.
Differential Revision: https://phabricator.services.mozilla.com/D74793
We had to increase the java buffer limit to 64MiB because it was a bottleneck
and 8MiB wasn't enough for 1ms sampling rate. It was getting full pretty quick.
Even though we increased that limit, we are not going to allocate all of this
at the start. Since we are allocating the buffer chunk by chunk, we are going
to allocate only when we need.
Differential Revision: https://phabricator.services.mozilla.com/D74715
If a label contains a dynamic string that's too long (512 characters or more), instead of just replacing it with "(too long)", we now cut it down to the maximum size, with an ellipsis at the end.
Added test for that in gtest. Also added nearby test for empty strings.
Differential Revision: https://phabricator.services.mozilla.com/D74378
Also removed explicit size in test_feature_nativeallocations, as we want a reasonably-big buffer in this test.
Other tests with small buffer sizes are fine, since the profiler imposes a minimum size that should be big enough to capture at least 8 large stacks (see bug 1571171).
Differential Revision: https://phabricator.services.mozilla.com/D74588
|./mach try| subcommands are now compatible with both python 2 and 3.
Hand-tested with many combinations of subcommand and subcommand flags.
Updates tryselect unit tests to use Python 3.
Differential Revision: https://phabricator.services.mozilla.com/D73398
This requires --build-peers-said-large-imports-were-ok since
third_party/rust/mp4parse/src/lib.rs is 113KB. This code is just moving from
media/mp4parse-rust to third_party/rust, so it's not really adding to net code
size.
Differential Revision: https://phabricator.services.mozilla.com/D74488
This change simply changes line endings to make diffing easier with updates from OpenVR, which have Unix line endings.
Further, .clang-format-ignore, does not seem to work for wildcard to match a directory and subdirectory, so all files are listed individually.
Differential Revision: https://phabricator.services.mozilla.com/D74100
These defaults were per process and there are usually around 8 processes.
Now these sizes apply to all processes, so they can be 8 times as big (but less on Android where memory may be more limited.)
Not changing Base Profiler defaults, as its buffer is not cross-process controlled.
Differential Revision: https://phabricator.services.mozilla.com/D73215
This implement the child side:
When the first request for update arrives, it connects to the local chunk manager, to receive its updates.
If multiple updates are received, they are folded into one.
If there are both an update and a pending request, the request is fulfilled with the update and local data is reset.
And ProfilerChild handles "destroy" instructions to destroy local chunks.
At this point, the whole machinery is in place, and all combined profile buffers used in all processes should use around the maximum amount allowed.
A bit more memory may still be used, e.g., due to IPC delays, and because of recycling which keeps some unused chunks alive for later reuse. But overall that should be a small amount compared to the usual user-requested limit.
Differential Revision: https://phabricator.services.mozilla.com/D72369
The logic part of the controller receives all updates, and makes decisions to destroy old chunks when the memory limit is reached.
Differential Revision: https://phabricator.services.mozilla.com/D72368
When there is at least one ProfilerParent (i.e., we are interacting with at least one child process) AND the parent profiler is running, the ProfilerParentTracker sets up the ProfileBufferGlobalController that will manage all chunks.
As a first step, it connects with the local chunk manager (to receive chunk updates), and sends update requests to all children.
(The actual controller logic is not implemented in this patch, nor is the ProfilerChild side, see following patches.)
Differential Revision: https://phabricator.services.mozilla.com/D72367
The Gecko Profiler can notify the ProfilerParent when it's about to stop (if it was started, but the notification will happen even when it's not started).
Differential Revision: https://phabricator.services.mozilla.com/D72365
The Gecko Profiler can provide its current controlled chunk manager.
It is the responsibility of the caller to keep track of the state of the profiler, to avoid using the chunk manager after it's discarded.
Differential Revision: https://phabricator.services.mozilla.com/D72364
The defaults should serve our users. I did some profiling with mochitests
and the defaults weren't ideal. It would be nice if the profiler "just worked"
for most of the situations. It would be great to teach the environment variables
about presets, but for now this will make the profiler easier to use for our
users in the general case.
This patch adds both screenshots and updates the entries count. This causes
additional overhead, but people on low-end devices can still tweak the settings
to remove that overhead.
Differential Revision: https://phabricator.services.mozilla.com/D70091
The second example, when copied and pasted, is completely borked since it
is not preceeded by the line that declares what follows to be a Mermaid
diagram. This change fixes that.
It also fixes the prose to read slightly less awkwardly in places.
Finally, this change also adds a link to the Mermaid Live Editor to make
readers aware of this useful tool.
Differential Revision: https://phabricator.services.mozilla.com/D74338
Closer look at the man pages shows that the file() query will catch any file
changes, whereas the modifies() query will only capture files that were
modified, which excludes strict deletions.
Depends on D74065
Differential Revision: https://phabricator.services.mozilla.com/D74066
profiler_is_locked_on_current_thread() is used to help users avoid calling `profiler_...` functions when the profiler may already have a lock in place, which would prevent a 2nd recursive lock (resulting in a crash or a never-ending wait).
So we must return `true` for any of:
- The main profiler mutex, used by most functions, and/or
- The buffer mutex, used directly in some functions without locking the main mutex, e.g., marker-related functions.
Differential Revision: https://phabricator.services.mozilla.com/D73789
At the time this code was written, JavaScript JIT entry trampolines were
emitting EnterJIT label frames that carried a stack address. From this stack
address, register values could be recovered that would allow native stack
unwinding to resume after getting lost in JIT code.
The EnterJIT label frame was removed in bug 1057082.
Differential Revision: https://phabricator.services.mozilla.com/D73939
In rare cases, a dispatch may fail (e.g., threads are shutting down), we should handle this failure by rejecting the promise on the spot.
Depends on D73791
Differential Revision: https://phabricator.services.mozilla.com/D73792
It is not necessary to dispatch another task to the main thread to resolve/reject a MozPromiseHolder, because wherever this happens, the attached `Then` will do its own dispatch to the main thread.
Also, this removes a dispatch that could potentially fail, leaving the promise unfulfilled.
Depends on D73790
Differential Revision: https://phabricator.services.mozilla.com/D73791
While gathering profiles from child processes, some actions (e.g., shutting down, restarting the profiler) may reset the gathering operation.
In this case we must ensure that the promise is rejected if not already fulfilled, so that anyone waiting on it won't be blocked forever (and MozPromise enforces it in a MOZ_DIAGNOSTIC_ASSERT).
Differential Revision: https://phabricator.services.mozilla.com/D73790
Code should be using `throw Components.Exception("", Cr.ERROR);` instead,
since `new Error()` just converts the int value of the Cr.ERROR into a string,
whereas `Exception` constructs an Exception object with the result property set
to the Cr.ERROR value, so other code can identify it.
Differential Revision: https://phabricator.services.mozilla.com/D28074
This rule is based on the ESLint built-in no-throw-literal. Cr.ERRORs are also
literals since they are just integers and so have all the same disadvantages of
no stack info.
TestInterfaceJS.js is explicitly testing handling of throwing raw Cr.ERRORs and
thus needs to stay.
Differential Revision: https://phabricator.services.mozilla.com/D28072
Raw Cr.ERROR don't get stack information, same as throwing JS literals instead
of `new Error()`s.
This was done automatically with a new eslint rule that will be introduced in
the next commit. One instance of a raw Cr.ERROR was not replaced since it is
used in a test that specifically checks the preservation of raw Cr values in
XPCJS. The rule will be disabled for that instance.
Differential Revision: https://phabricator.services.mozilla.com/D28073
The profiler can be given any power of two, but there were no safety checks to ensure that the buffer had a minimum workable capacity (to hold at least 4 chunks, each capable of holding at least one stack sample), and also to prevent large buffers that could break the currently-supported 2GiB limit.
This fixes the issue with test_merged_stacks.js, which was requesting a too-small buffer. (This started when we switched to the chunk-based buffer, because the profiler was blindly using the provided number as maximum, and dividing that size by 4 for each chunk, which was not enough to hold a full sample in some builds.)
Differential Revision: https://phabricator.services.mozilla.com/D73212
This was an issue around "six" and "click", this thread sheds some light:
https://github.com/pallets/click/issues/564.
This issue was reproducible by having a local python 2 flask app with Flask==1.0.2, Werkzeug==0.14.1 and Click==7.0. Upon self-shutdown, the TypeError was raised.
Upgrading Click to 7.1.2 resolves this issue both in that separate "reproduce app" and here in-tree.
Differential Revision: https://phabricator.services.mozilla.com/D73351
Gerald Squelart
Jean-Yves Avenard
Kris Maglione
Simon Fraser
Mike Hommey
Nazım Can Altınova
Matt Woodrow
Jared Wein
Gijs Kruitbosch
Kartikaya Gupta
Dorel Luca
Sylvestre Ledru
Sonia Singla
Narcis Beleuzu
Matthew Gregan
Ricky Stewart
Simon Sapin
Chris AtLee
Ted Campbell
Csoregi Natalia
Henri Sivonen
Paul Adenot
Myeongjun Go
Razvan Maries
Mitchell Hentges
Adam Gashlin
Simon Giesecke
Andrew Halberstadt
championshuttler
Tom Ritter
Gregory Mierzwinski
Andrew McCreight
Christian Holler
Bob Clary
Nicolas Silva
Agi Sferro
Mark Smith
Bogdan Tara
Geoff Brown
nchevobbe
Nicholas Nethercote
Greg Tatum
Toshihito Kikuchi
Dzmitry Malyshau
Markus Stange
Kershaw Chang
Butkovits Atila
Tilden Windsor
Tom Prince
Coroiu Cristina
Marco Castelluccio
alwu
Aaron Klotz
Edouard Oger
Tarek Ziadé
Mihai Alexandru Michis
Jon Bauman
Chris Fronk
thomasmo
Jonathan Watt
Cosmin Sabou
Greg V
Ian Moody
Julien Wajsberg
undef1nd
Michael Hoffmann
Matthew Gaudet
David Teller
Stefan Hindli