Unique strings are used to encode all markers' 'name' field, SpliceableJSONWriter::UniqueStringElement can be used for that (instead of a caller-provided callback, which was necessary before UniqueJSONStrings was de-duplicated).
Differential Revision: https://phabricator.services.mozilla.com/D95513
Some markers (e.g., Screenshot) use unique strings in their data.
The UniqueJSONStrings used during streaming is attached to the SpliceableJSONWriter, and StreamJSONMarkerData can use pass-through functions UniqueStringProperty() and UniqueStringElement().
Differential Revision: https://phabricator.services.mozilla.com/D95512
Some markers (e.g., GC major/minor/slice) need to splice JSON strings in their data.
So now, instead of JSONWriter, StreamJSONMarkerData functions will be given a mozilla::baseprofiler::SpliceableJSONWriter.
Differential Revision: https://phabricator.services.mozilla.com/D95511
`UniqueJSONStrings` constructors now accept a `JSONWriter::CollectionStyle` to pass to the internal JSON writer.
The default won't change how the unique string list is output (with friendly indented lines), but tests can use `JSONWriter::SingleLineStyle` to make comparison strings more readable and maintainable.
Differential Revision: https://phabricator.services.mozilla.com/D95832
One of the scenarios in TestMMPolicy.exe is to reserve all regions but one free
region, and then FindRegion can find that free region. However, it intermittently
failed to find a free region on 32bit. Probably a different thread invoked by
the system reserved it before the main thread does. A proposed fix is to limit
the address range to reserve.
Differential Revision: https://phabricator.services.mozilla.com/D95188
The latest launcher process showed one of the top failures was `WriteProcessMemory` in
`CopyStubToChildProcess` failed with `ERROR_INVALID_ADDRESS` or `ERROR_NOACCESS`, that
is to store a trampoline address to the global variable of firefox.exe failed. Its root
cause should be the same as bug 1662560, the executable was loaded onto a different
address from the browser process.
The fix is to to expand the usage of `CrossExecTransferManager` to `FuncHookCrossProcess`
and `Kernel32ExportsSolver`.
Depends on D94652
Differential Revision: https://phabricator.services.mozilla.com/D94653
This commit uses the new Markers 2.0 API for FileIO Markers. I had to
create another option for the MarkerStack class in order to conditionally
capture a backtrace inside of the Macro. Otherwise the macro invocation
failed.
Differential Revision: https://phabricator.services.mozilla.com/D93848
This commit uses the new Markers 2.0 API for FileIO Markers. I had to
create another option for the MarkerStack class in order to conditionally
capture a backtrace inside of the Macro. Otherwise the macro invocation
failed.
Differential Revision: https://phabricator.services.mozilla.com/D93848
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
The latest launcher process ping showed one of the reasons why we failed to
detour `NtMapViewOfSection` is that `MMPolicyBase::FindRegion` failed to find
a free region. Inspecting the function carefully, there were three problems.
Firstly, `FindRegion` did not fully scan the given range. To randomize
the address of a free region we use, we start scanning from a random address
within the given range. The problem is we scan only addresses bigger than
that random address, without scanning smaller addresses. Probably this is
the reason why `FindRegion` fails.
Secondly, `FindRegion` may return an address not aligned with the allocation
granularity because `VirtualQueryEx` returns such an address. If that happens,
the subsequent mapping API fails with the alignment error.
Lastly, when we randomize an address to start scanning from, we divide a random
number by `maxOffset`, but with that, we never start scanning from the last
region. It does not affect the product's behavior, but to have fair randomization,
a divisor should be `maxOffset + 1`.
This patch fixes all of these three problems along with a new test program.
Differential Revision: https://phabricator.services.mozilla.com/D94110
Everything related to Base Profiler legacy markers can now be removed, only the new API from BaseProfilerMarkers.h should now be used.
Depends on D93737
Differential Revision: https://phabricator.services.mozilla.com/D93738
Since the main thread is almost always being profiled, this makes it easy to direct important markers there (e.g., FileIO markers from unregistered threads).
Tech note: The `#include "BaseProfiler.h"` line was moved to BaseProfilerMarkersPrerequesites.h so that `MarkerThreadId::MainThread()` there can access `profiler_main_thread_id()`.
Depends on D93735
Differential Revision: https://phabricator.services.mozilla.com/D93736
These functions will be useful to get the main thread id, or check if we're in it, in some public code (e.g., markers).
Differential Revision: https://phabricator.services.mozilla.com/D93735
These unit tests were going through chunks very quickly in tight loops, which on some platforms could cause successive chunks to be marked "done" with the same timestamps.
Distinct timestamps are needed to uniquely identify chunks (in each process), and related assertions caught that. In real Firefox code, chunks are much bigger (around 1MB), and fill at a slower rate, so timestamps are always different as expected there.
To fix this artificial issue in our unit tests, delays are introduced before each `MarkDone()`, to ensure that they'll get a different timestamp every time.
Differential Revision: https://phabricator.services.mozilla.com/D93479
Add `static mozilla::MarkerSchemaWriter MarkerTypeDisplay()` for each existing marker type.
Because all markers of a given type now must have the same payload and display schema, the DOM event marker has changed from type=tracing with category=DOMEvent, to its own type=DOMEvent, which is now accepted on the front-end.
Based on c9692715f2/src/profile-logic/marker-schema.js
Differential Revision: https://phabricator.services.mozilla.com/D90658
This is the third attempt to investigate the launcher failure of our detour.
The previous commits d8315e4ed18d and 1b81ea85c43d added the assembly bytes
of a detour target and a special error code `DetourResultCode` to the launcher
failure ping.
In the latest telemetry data, however, the most common value of `hresult`
is still `ERROR_UNIDENTIFIED_ERROR`, meaning the previous commit missed to
set an error code in the common fallible codepath we wanted to know.
Besides `ERROR_UNIDENTIFIED_ERROR`, we're seeing `DETOUR_PATCHER_DO_RESERVE_ERROR`
in the telemetry, but having that code is not enough to pinpoint a falling
operation.
For further investigation, this patch adds ten more values to `DetourResultCode`.
`FUNCHOOKCROSSPROCESS_COPYSTUB_ERROR` is the last codepath we forgot to cover
in the previous commit. The values of `MMPOLICY_RESERVE_*` are to investigate
`DETOUR_PATCHER_DO_RESERVE_ERROR` in the MMPolicy level. In both cases, we add
the last Windows error code to `DetourError::mOrigBytes`.
Differential Revision: https://phabricator.services.mozilla.com/D92974
The latest Windows Insider Preview (version 20226.1000) changes the machine code for BaseThreadInitThunk to have a preamble like the following:
00007FFDBF244C40 48 83 EC 28 sub rsp,28h
00007FFDBF244C44 85 C9 test ecx,ecx
00007FFDBF244C46 75 25 jne 00007FFDBF244C6D
00007FFDBF244C48 49 BA 70 A2 DC 12 6A 97 99 B0 mov r10,0B099976A12DCA270h
This patch adds "MOV r64, imm64" capability to the DLL interceptor so that we can hook this.
Differential Revision: https://phabricator.services.mozilla.com/D92146
This makes it clearer where marker-type-specific payload arguments start, just after the marker type object.
Also improved the main API documentation.
Differential Revision: https://phabricator.services.mozilla.com/D91681
The `category.WithOptions(...)` syntax was a bit strange and difficult to explain.
Now the category and options are separate parameters. Default options can be specified with `MarkerOptions{}` or just `{}`.
As a special case, defaulted-NoPayload functions don't need `<>`, and defaulted-NoPayload functions and macros don't even need `{}` for default options, e.g.:
`profiler_add_marker("name", OTHER); PROFILER_MARKER_UNTYPED("name", OTHER);`
Differential Revision: https://phabricator.services.mozilla.com/D91680
This makes it clearer where marker-type-specific payload arguments start, just after the marker type object.
Also improved the main API documentation.
Differential Revision: https://phabricator.services.mozilla.com/D91681
The `category.WithOptions(...)` syntax was a bit strange and difficult to explain.
Now the category and options are separate parameters. Default options can be specified with `MarkerOptions{}` or just `{}`.
As a special case, defaulted-NoPayload functions don't need `<>`, and defaulted-NoPayload functions and macros don't even need `{}` for default options, e.g.:
`profiler_add_marker("name", OTHER); PROFILER_MARKER_UNTYPED("name", OTHER);`
Differential Revision: https://phabricator.services.mozilla.com/D91680
The last Avast Antivirus's hook function contains `CALL [disp32]` instruction.
Our detour needs to be able to handle that pattern.
Differential Revision: https://phabricator.services.mozilla.com/D91155
This patch optimizes our detour's code handling Opcode 0xFF, expanding
its coverage to INC and DEC reg64 as well as PUSH and CALL.
Testcases for these scenarios are of course included.
Differential Revision: https://phabricator.services.mozilla.com/D91154
This saves 1 byte when serializing each marker (and removes all the code that was related to the 2nd byte).
Also it will be easier to use it in legacy code that only knows about the category pair.
Added unit tests for the whole of MarkerCategory.
Differential Revision: https://phabricator.services.mozilla.com/D91110
In most situations, JSONWriter users already know string lengths (either directly, or through `nsCString` and friends), so we should keep this information through JSONWriter and not recompute it again.
This also allows using JSONWriter with sub-strings (e.g., from a bigger buffer), without having to create null-terminated strings.
Public JSONWriter functions have overloads that accept literal strings.
Differential Revision: https://phabricator.services.mozilla.com/D86192
The name `AUTO_PROFILER_MARKER_TEXT` is more consistent with the equivalent non-`AUTO` macro, and similarly arguments have been re-ordered to be the same, i.e.: Name, category&options, text.
The different macros with different argument sets can now be collapsed into one macro, and the optional arguments (timing, inner window id, backtrace) can easily be added to the `MarkerOptions` where needed.
As a bonus, a specific start time can optionally be provided at construction time.
Differential Revision: https://phabricator.services.mozilla.com/D89588
Mostly mechanical change, with some extra work where non-literal names are provided.
Also, when this is the only profiler call in a file, `#include "GeckoProfiler.h"` can be changed to `#include "mozilla/ProfilerMarkers.h"`.
Differential Revision: https://phabricator.services.mozilla.com/D89415
`ProfileChunkedBuffer` needed to be fully defined, because its destructor is needed to define `UniquePtr<ProfileChunkedBuffer>`.
It can just be empty, because it won't actually be used anyway.
Added non-`MOZ_GECKO_PROFILER` tests around this.
Differential Revision: https://phabricator.services.mozilla.com/D89351
These string views are similar to `std::string_view`, but they are optimized to be serialized in the profiler buffer, and later deserialized and streamed to JSON.
They accept literal strings, and keep them as unowned raw pointers and sizes.
They also accept any substring reference, assuming that they will only be used as parameters during function calls, and therefore the dependent string will live during that call where these `StringView`'s are used.
Internally, they also allow optional string ownership, which is only used during deserialization and streaming.
This is hidden, so that users are not tempted to use potentially expensive string allocations during profiling; it's only used *after* profiling, so it's less of an impact to allocate strings then. (But it could still be optimized later on, as part of bug 1577656.)
Differential Revision: https://phabricator.services.mozilla.com/D87242
These string views are similar to `std::string_view`, but they are optimized to be serialized in the profiler buffer, and later deserialized and streamed to JSON.
They accept literal strings, and keep them as unowned raw pointers and sizes.
They also accept any substring reference, assuming that they will only be used as parameters during function calls, and therefore the dependent string will live during that call where these `StringView`'s are used.
Internally, they also allow optional string ownership, which is only used during deserialization and streaming.
This is hidden, so that users are not tempted to use potentially expensive string allocations during profiling; it's only used *after* profiling, so it's less of an impact to allocate strings then. (But it could still be optimized later on, as part of bug 1577656.)
Differential Revision: https://phabricator.services.mozilla.com/D87242
These string views are similar to `std::string_view`, but they are optimized to be serialized in the profiler buffer, and later deserialized and streamed to JSON.
They accept literal strings, and keep them as unowned raw pointers and sizes.
They also accept any substring reference, assuming that they will only be used as parameters during function calls, and therefore the dependent string will live during that call where these `StringView`'s are used.
Internally, they also allow optional string ownership, which is only used during deserialization and streaming.
This is hidden, so that users are not tempted to use potentially expensive string allocations during profiling; it's only used *after* profiling, so it's less of an impact to allocate strings then. (But it could still be optimized later on, as part of bug 1577656.)
Differential Revision: https://phabricator.services.mozilla.com/D87242
These string views are similar to `std::string_view`, but they are optimized to be serialized in the profiler buffer, and later deserialized and streamed to JSON.
They accept literal strings, and keep them as unowned raw pointers and sizes.
They also accept any substring reference, assuming that they will only be used as parameters during function calls, and therefore the dependent string will live during that call where these `StringView`'s are used.
Internally, they also allow optional string ownership, which is only used during deserialization and streaming.
This is hidden, so that users are not tempted to use potentially expensive string allocations during profiling; it's only used *after* profiling, so it's less of an impact to allocate strings then. (But it could still be optimized later on, as part of bug 1577656.)
Differential Revision: https://phabricator.services.mozilla.com/D87242
The main change is removing ProfileJSONWriter.cpp, making ProfileJSONWriter.h point at BaseProfileJSONWriter.h, and exposing `mozilla::baseprofiler::` classes in the top namespace as expected by users of ProfileJSONWriter.h (to minimize changes).
These two headers are now always present in the "mozilla" include directory, independent of MOZ_GECKO_PROFILER settings.
The rest is just needed tweaks to match the above changes.
Differential Revision: https://phabricator.services.mozilla.com/D86504
Gerald Squelart
Toshihito Kikuchi
Greg Tatum
Butkovits Atila
Ricky Stewart
Bogdan Tara
Dorel Luca
David Parks
Cristina Coroiu
Brindusan Cristian
Narcis Beleuzu
Simon Giesecke