If EAF+ is enabled for firefox.exe, the process does not launch because we parse
the PE headers of ntdll.dll at startup, which is prohibited by EAF+.
With this patch, we skip two operations when EAF+ is enabled.
The first one is to cache ntdll's IAT at startup. Because EAF+ is expected to
prevent an injected module from parsing PE headers and modifying IAT, we can skip
this caching safely.
The second one is to load ntdll's debug information for the profiler. With this
patch, the profiler's callstack will not show a raw address instead of a symbol
name. It's a bad side effect, but much better than startup crash.
Differential Revision: https://phabricator.services.mozilla.com/D76959
This is the logic of tracing the WebGPU API calls at the level of wgpu-core,
serialized into a folder of choosing on the user drive. Traces are extremely portable,
they can be shared (on BugZilla) and then replayed on the developer machine,
which can have a different architecture from the users machine.
The standalone player is introduced in `gfx/wgpu/player`, similar to WebRender's Wrench.
The output dir is controlled by "dom.webgpu.traceDir" pref. No tracing happens if it's empty.
Differential Revision: https://phabricator.services.mozilla.com/D73333
This is the logic of tracing the WebGPU API calls at the level of wgpu-core,
serialized into a folder of choosing on the user drive. Traces are extremely portable,
they can be shared (on BugZilla) and then replayed on the developer machine,
which can have a different architecture from the users machine.
The standalone player is introduced in `gfx/wgpu/player`, similar to WebRender's Wrench.
The output dir is controlled by "dom.webgpu.traceDir" pref. No tracing happens if it's empty.
Differential Revision: https://phabricator.services.mozilla.com/D73333
This code was not being built any more.
GeckoView requires Android OS version 4.3 (API Level 18). I'm not sure what the
corresponding ANDROID_VERSION macro constant for API Level 18 is, but it's
definitely higher than 16.
Differential Revision: https://phabricator.services.mozilla.com/D73940
Also change the processor IDs to hex, because that's what the Linux kernel
source code uses. And reorder the cases a little.
Differential Revision: https://phabricator.services.mozilla.com/D76272
When the tracker is destroyed, any extant channel is closed, this can trigger the rejection of pending promises, which dispatch tasks to promise handlers. So we need to ensure that this happens before threads cannot service dispatches anymore.
Differential Revision: https://phabricator.services.mozilla.com/D76436
In addition to the existing "mainthreadio" feature, we now have:
- "fileio" to also capture file I/O from other profiled threads.
- "fileioall" to also capture file I/O from all threads (even unregistered threads).
- "noiostacks" to prevent capturing stack traces for "io" markers.
These are all off by default, except for `MOZ_PROFILER_STARTUP=1`.
Differential Revision: https://phabricator.services.mozilla.com/D75764
This can be used while the profiler is running, to know if the current thread is registered -- regardless of whether it is actively profiled.
This will help distinguish registered but non-profiled threads from threads that are not even registered (e.g., OS-generated threads).
Differential Revision: https://phabricator.services.mozilla.com/D76281
Some functions may want to examine more than one feature at once, these function return everything (if the profiler is active, and optionally unpaused) so that only one call is needed.
Differential Revision: https://phabricator.services.mozilla.com/D76280
This is a simple compile-time check that the feature numbers go from 0 to N-1, to prevent some mistakes when updating feature lists.
Differential Revision: https://phabricator.services.mozilla.com/D75763
The FileIO marker may be recorded into the main thread, so we need to keep track of the thread in which the IO really happened (which may be a thread that's not even selected for profiling).
There is not JSON output for it yet.
Differential Revision: https://phabricator.services.mozilla.com/D75761
Some markers may be more useful when gathered into a single track, and the main thread is ideal because it's almost always present.
Differential Revision: https://phabricator.services.mozilla.com/D75760
I need to add symbolication support for the mochitest Gecko Profiler command line
option. These profiles also need to be symbolicated. Unfortunately, there is not
a common place where I could use these files. Talos and Raptor each had their
own copy of the snappy symbolication server.
This commit consolidates these packages into a re-usable mozbase package that can
be used in mochitests, and eventually in other places like xpcshell tests.
I stubbed out a test file, but it doesn't do anything quite yet. This commit makes
it so that the tests still work in Raptor and Talos, but doesn't add any features.
It also doesn't try too hard to make the files look like a mozbase package.
Differential Revision: https://phabricator.services.mozilla.com/D74289
Instead of accepting a `UniquePtr<ProfilerMarkerPayload>`, `profiler_add_marker_for_thread` now takes it by `const ProfilerMarkerPayload&`.
All callers can now create the payload on the stack, or as a temporary object. This saves a memory alloc+free for each call.
Differential Revision: https://phabricator.services.mozilla.com/D75912
This patch will do :
- add `Media` markers in related codes
- implement `MediaSampleMarkerPayload` to record the media sample
The advantage of doing so :
- using markers can help us know what happens on the media by a glance without expanding the call stack
- adding sample markers allows us compare the speed of decoding sample in `MediaDecoderStataMachine` and rendering sample in `VideoSink`
Differential Revision: https://phabricator.services.mozilla.com/D74174
In this bug we're moving away from monolithic JNI headers to class-specific
headers so that we don't have to rebuild the world every time we make a change
to a JNI interface.
Differential Revision: https://phabricator.services.mozilla.com/D75368
This patch is adding an option to push a perftest run in the CI.
It's based on :
- sparse profiles
- push_to_try
- options passed through try_task_config.json
Differential Revision: https://phabricator.services.mozilla.com/D74115
Mach doesn't know which tasks are part of a release - that's decided in the decision task. Instead of
showing a bogus estimation, we shouldn't show one at all.
Differential Revision: https://phabricator.services.mozilla.com/D74904
In some NT-specific code, a list of "items()" was being updated. In python 3, to modify the result
of "items()", you have to gain ownership first by explicitly converting it into a list().
Also resolves some flaky failures that were seen locally by marking test_presets.py as sequential.
Differential Revision: https://phabricator.services.mozilla.com/D74793
We had to increase the java buffer limit to 64MiB because it was a bottleneck
and 8MiB wasn't enough for 1ms sampling rate. It was getting full pretty quick.
Even though we increased that limit, we are not going to allocate all of this
at the start. Since we are allocating the buffer chunk by chunk, we are going
to allocate only when we need.
Differential Revision: https://phabricator.services.mozilla.com/D74715
If a label contains a dynamic string that's too long (512 characters or more), instead of just replacing it with "(too long)", we now cut it down to the maximum size, with an ellipsis at the end.
Added test for that in gtest. Also added nearby test for empty strings.
Differential Revision: https://phabricator.services.mozilla.com/D74378
Also removed explicit size in test_feature_nativeallocations, as we want a reasonably-big buffer in this test.
Other tests with small buffer sizes are fine, since the profiler imposes a minimum size that should be big enough to capture at least 8 large stacks (see bug 1571171).
Differential Revision: https://phabricator.services.mozilla.com/D74588
|./mach try| subcommands are now compatible with both python 2 and 3.
Hand-tested with many combinations of subcommand and subcommand flags.
Updates tryselect unit tests to use Python 3.
Differential Revision: https://phabricator.services.mozilla.com/D73398
This requires --build-peers-said-large-imports-were-ok since
third_party/rust/mp4parse/src/lib.rs is 113KB. This code is just moving from
media/mp4parse-rust to third_party/rust, so it's not really adding to net code
size.
Differential Revision: https://phabricator.services.mozilla.com/D74488
This change simply changes line endings to make diffing easier with updates from OpenVR, which have Unix line endings.
Further, .clang-format-ignore, does not seem to work for wildcard to match a directory and subdirectory, so all files are listed individually.
Differential Revision: https://phabricator.services.mozilla.com/D74100
These defaults were per process and there are usually around 8 processes.
Now these sizes apply to all processes, so they can be 8 times as big (but less on Android where memory may be more limited.)
Not changing Base Profiler defaults, as its buffer is not cross-process controlled.
Differential Revision: https://phabricator.services.mozilla.com/D73215
This implement the child side:
When the first request for update arrives, it connects to the local chunk manager, to receive its updates.
If multiple updates are received, they are folded into one.
If there are both an update and a pending request, the request is fulfilled with the update and local data is reset.
And ProfilerChild handles "destroy" instructions to destroy local chunks.
At this point, the whole machinery is in place, and all combined profile buffers used in all processes should use around the maximum amount allowed.
A bit more memory may still be used, e.g., due to IPC delays, and because of recycling which keeps some unused chunks alive for later reuse. But overall that should be a small amount compared to the usual user-requested limit.
Differential Revision: https://phabricator.services.mozilla.com/D72369
The logic part of the controller receives all updates, and makes decisions to destroy old chunks when the memory limit is reached.
Differential Revision: https://phabricator.services.mozilla.com/D72368
When there is at least one ProfilerParent (i.e., we are interacting with at least one child process) AND the parent profiler is running, the ProfilerParentTracker sets up the ProfileBufferGlobalController that will manage all chunks.
As a first step, it connects with the local chunk manager (to receive chunk updates), and sends update requests to all children.
(The actual controller logic is not implemented in this patch, nor is the ProfilerChild side, see following patches.)
Differential Revision: https://phabricator.services.mozilla.com/D72367
The Gecko Profiler can notify the ProfilerParent when it's about to stop (if it was started, but the notification will happen even when it's not started).
Differential Revision: https://phabricator.services.mozilla.com/D72365
The Gecko Profiler can provide its current controlled chunk manager.
It is the responsibility of the caller to keep track of the state of the profiler, to avoid using the chunk manager after it's discarded.
Differential Revision: https://phabricator.services.mozilla.com/D72364
The defaults should serve our users. I did some profiling with mochitests
and the defaults weren't ideal. It would be nice if the profiler "just worked"
for most of the situations. It would be great to teach the environment variables
about presets, but for now this will make the profiler easier to use for our
users in the general case.
This patch adds both screenshots and updates the entries count. This causes
additional overhead, but people on low-end devices can still tweak the settings
to remove that overhead.
Differential Revision: https://phabricator.services.mozilla.com/D70091
The second example, when copied and pasted, is completely borked since it
is not preceeded by the line that declares what follows to be a Mermaid
diagram. This change fixes that.
It also fixes the prose to read slightly less awkwardly in places.
Finally, this change also adds a link to the Mermaid Live Editor to make
readers aware of this useful tool.
Differential Revision: https://phabricator.services.mozilla.com/D74338
Closer look at the man pages shows that the file() query will catch any file
changes, whereas the modifies() query will only capture files that were
modified, which excludes strict deletions.
Depends on D74065
Differential Revision: https://phabricator.services.mozilla.com/D74066
profiler_is_locked_on_current_thread() is used to help users avoid calling `profiler_...` functions when the profiler may already have a lock in place, which would prevent a 2nd recursive lock (resulting in a crash or a never-ending wait).
So we must return `true` for any of:
- The main profiler mutex, used by most functions, and/or
- The buffer mutex, used directly in some functions without locking the main mutex, e.g., marker-related functions.
Differential Revision: https://phabricator.services.mozilla.com/D73789
At the time this code was written, JavaScript JIT entry trampolines were
emitting EnterJIT label frames that carried a stack address. From this stack
address, register values could be recovered that would allow native stack
unwinding to resume after getting lost in JIT code.
The EnterJIT label frame was removed in bug 1057082.
Differential Revision: https://phabricator.services.mozilla.com/D73939
In rare cases, a dispatch may fail (e.g., threads are shutting down), we should handle this failure by rejecting the promise on the spot.
Depends on D73791
Differential Revision: https://phabricator.services.mozilla.com/D73792
It is not necessary to dispatch another task to the main thread to resolve/reject a MozPromiseHolder, because wherever this happens, the attached `Then` will do its own dispatch to the main thread.
Also, this removes a dispatch that could potentially fail, leaving the promise unfulfilled.
Depends on D73790
Differential Revision: https://phabricator.services.mozilla.com/D73791
While gathering profiles from child processes, some actions (e.g., shutting down, restarting the profiler) may reset the gathering operation.
In this case we must ensure that the promise is rejected if not already fulfilled, so that anyone waiting on it won't be blocked forever (and MozPromise enforces it in a MOZ_DIAGNOSTIC_ASSERT).
Differential Revision: https://phabricator.services.mozilla.com/D73790
Code should be using `throw Components.Exception("", Cr.ERROR);` instead,
since `new Error()` just converts the int value of the Cr.ERROR into a string,
whereas `Exception` constructs an Exception object with the result property set
to the Cr.ERROR value, so other code can identify it.
Differential Revision: https://phabricator.services.mozilla.com/D28074
This rule is based on the ESLint built-in no-throw-literal. Cr.ERRORs are also
literals since they are just integers and so have all the same disadvantages of
no stack info.
TestInterfaceJS.js is explicitly testing handling of throwing raw Cr.ERRORs and
thus needs to stay.
Differential Revision: https://phabricator.services.mozilla.com/D28072
Raw Cr.ERROR don't get stack information, same as throwing JS literals instead
of `new Error()`s.
This was done automatically with a new eslint rule that will be introduced in
the next commit. One instance of a raw Cr.ERROR was not replaced since it is
used in a test that specifically checks the preservation of raw Cr values in
XPCJS. The rule will be disabled for that instance.
Differential Revision: https://phabricator.services.mozilla.com/D28073
The profiler can be given any power of two, but there were no safety checks to ensure that the buffer had a minimum workable capacity (to hold at least 4 chunks, each capable of holding at least one stack sample), and also to prevent large buffers that could break the currently-supported 2GiB limit.
This fixes the issue with test_merged_stacks.js, which was requesting a too-small buffer. (This started when we switched to the chunk-based buffer, because the profiler was blindly using the provided number as maximum, and dividing that size by 4 for each chunk, which was not enough to hold a full sample in some builds.)
Differential Revision: https://phabricator.services.mozilla.com/D73212
This was an issue around "six" and "click", this thread sheds some light:
https://github.com/pallets/click/issues/564.
This issue was reproducible by having a local python 2 flask app with Flask==1.0.2, Werkzeug==0.14.1 and Click==7.0. Upon self-shutdown, the TypeError was raised.
Upgrading Click to 7.1.2 resolves this issue both in that separate "reproduce app" and here in-tree.
Differential Revision: https://phabricator.services.mozilla.com/D73351
Because the web platform test harness doesn't recognize them and annotates them
with `CRITICAL`, which is disconcerting, even though those errors are typically
not harmful.
Differential Revision: https://phabricator.services.mozilla.com/D71485
This is how my mapfile looks like:
5587df936000-5587df96b000 r--p 00000000 fd:02 21584889 /home/emilio/src/moz/gecko/obj-debug/dist/bin/firefox
5587df96b000-5587df9ec000 r-xp 00035000 fd:02 21584889 /home/emilio/src/moz/gecko/obj-debug/dist/bin/firefox
5587df9ec000-5587df9ed000 r--p 000b6000 fd:02 21584889 /home/emilio/src/moz/gecko/obj-debug/dist/bin/firefox
5587df9ed000-5587df9ee000 rw-p 000b7000 fd:02 21584889 /home/emilio/src/moz/gecko/obj-debug/dist/bin/firefox
Note how the executable bit, which is the only one we look at, and which
is where we get exeExeAddr from (0x5587df96b000 in this case) is in the
middle of the executable, but the library will span all four ranges.
Check for whether the library _contains_ the start address of the
executable region instead of whether it starts with it.
Differential Revision: https://phabricator.services.mozilla.com/D72504
Renamed some variables to be more generic. Their type is going to change in the next patch, and that type doesn't need to be in the names; also it will make the next patch easier to review.
Differential Revision: https://phabricator.services.mozilla.com/D71500
Instead of a potentially-null pointer to a `ProfileBufferEntryWriter`, we are now providing a `Maybe<ProfileBufferEntryWriter>`, which is safer.
Differential Revision: https://phabricator.services.mozilla.com/D71286
This ensures we don't run every build with every push via ./mach try auto. It
introduces a new 'optimization-overrides' try_config that can be used to
replace optimizations. For now, there is no user interface to pass this in via
the 'mach try' command line.
Differential Revision: https://phabricator.services.mozilla.com/D68207
Changes:
Applies the `filter_tasks_by_blacklist` method to try syntax pushes as well.
- moved `TARGET_TASK_BLACKLIST`and `filter_tasks_by_blacklist` method to live in `taskcluster/taskgraph/target_tasks.py`.
- removed existing filters against `ccov, windows10-aarch64` and `android-hw` filters against try syntax pushes.
- update imports for `fuzzy` and `chooser` selectors to refer to the new location of `filter_tasks_by_blacklist` method.
The reason for moving the logic (again) from `tools/tryselect` to `taskcluster/` is due to the placement of `try_option_syntax` and `target_tasks` files and both of those files handle the processing of `mach try syntax` pushes.
Differential Revision: https://phabricator.services.mozilla.com/D71698
This commit:
- removes sendRpcMessage, which was unused;
- removes the CPOW argument to sendAsyncMessage, broadcastAsyncMessage, and
sendSyncMessage;
- removes the aIsSync argument used internally to distinguish sendRpcMessage
and sendSyncMessage;
- removes CPOW tests;
- updates the few remaining callsites that use more than 2 arguments in
sendAsyncMessage for the removal of the cpows argument.
Differential Revision: https://phabricator.services.mozilla.com/D71514
Currently AWSY-with-DMD doesn't work on Windows. This is because `fix-stacks`
is initialized lazily, and by the time the initialization happens some file
descriptors for files are open, and that leads to some major Python2-on-Windows
sadness as described in the big comment in the commit.
To fix the problem, this commit adds an `init` function to `fix_stacks.py` so
that `fix-stacks` can be initialized eagerly, hopefully before any file
descriptors for files are open.
For `dmd.py`, other than fixing the AWSY problems, this has little effect,
because `fix-stacks` is always initialized.
For `utils.py`, which is used to process the output of most tests, this has a
more noticeable effect: the `fix-stacks` process is always spawned, rather than
being spawned only when needed. If no stack traces appear in the test output,
this means that `fix-stacks` is spawned unnecessarily. But it's cheap to spawn;
the expensive part only happens when stack traces start getting fixed. So I
think this change in behaviour is acceptable.
Furthermore, the commit adds a `finish` function to `fix_stacks.py`, so that
the `fix-stacks` process can be explicitly shut down. This has never been done
for processes spawned for any of the stack fixing scripts. It's never caused
problems on Linux/Mac, but it seems to be necessary on Windows to avoid
similar "this file is locked" problems with the test_dmd.js test.
The commit also renames some things to more standard Python style, e.g.
`json_mode` instead of `jsonMode`.
Finally, Android tests use `utils.py` from the repository but `fix_stacks.py`
from the Android host utils. Because the two scripts must be updated in tandem,
this commit also updates the Android host utils to a version that contains the
updated `fix_stacks.py`. Thanks to aerickson for packaging up the new Android
host utils and providing the change to the `hostutils.manifest` file.
Differential Revision: https://phabricator.services.mozilla.com/D69478
For now, the local detection sucks. I will fix that once bug 1625884
is fixed
Differential Revision: https://phabricator.services.mozilla.com/D69683
--HG--
extra : moz-landing-system : lando
`ProfileBuffer` used to check if a `BlocksRingBuffer` was in session by looking
at its buffer size.
Now `IsInSession()` should be used, it is clearer in intent, and will make the
transition to `ProfileChunkedBuffer` slightly easier.
Differential Revision: https://phabricator.services.mozilla.com/D69492
--HG--
extra : moz-landing-system : lando
Changes:
- implement filtering of taskgraph items against a list of known task filters in `tasks.py`.
- apply these filters to both `fuzzy` and `chooser` selectors, unless `--full` is specified.
Differential Revision: https://phabricator.services.mozilla.com/D70097
--HG--
extra : moz-landing-system : lando
This commit removes `test_fix_stack_using_bpsyms.py`. That test can't easily be
modified to work with `fix_stacks.py` because it relies on internal
implementation details of `fix_stack_using_bpsym.py`. The unit testing done in
the `fix-stacks` repo provides test coverage that is as good or better.
Differential Revision: https://phabricator.services.mozilla.com/D66924
--HG--
extra : moz-landing-system : lando
Changes:
- implement filtering of taskgraph items against a list of known task filters in `tasks.py`.
- apply these filters to both `fuzzy` and `chooser` selectors, unless `--full` is specified.
Differential Revision: https://phabricator.services.mozilla.com/D70097
--HG--
extra : moz-landing-system : lando
Since the test goes through all moz.build files disregarding DIRS and
the conditions that may disable directories, in some cases, moz.builds
can fail to be evaluated properly because of missing variables in
config.status. This time (because it's not the first), it's
LLVM_DLLTOOL.
After fixing that, it turns out many of the files/directories pointed to
by Files() directives were removed or moved.
While here, make the test script python3-ready.
Differential Revision: https://phabricator.services.mozilla.com/D70157
--HG--
extra : moz-landing-system : lando
Also move MOZ_MUST_USE before function declarations' specifiers and return type. While clang and gcc's __attribute__((warn_unused_result)) can appear before, between, or after function specifiers and return types, the [[nodiscard]] attribute must precede the function specifiers.
Differential Revision: https://phabricator.services.mozilla.com/D69739
--HG--
extra : moz-landing-system : lando
Update code in a few places to work with Python 3, especially with regard to how we invoke subprocesses with text output. Note that the Marionette test harness code needs to be updated so that text is written out in the correct encoding (see bug 1615748).
Differential Revision: https://phabricator.services.mozilla.com/D69012
--HG--
extra : moz-landing-system : lando
Currently AWSY-with-DMD doesn't work on Windows. This is because `fix-stacks`
is initialized lazily, and by the time the initialization happens some file
descriptors for files are open, and that leads to some major Python2-on-Windows
sadness as described in the big comment in the commit.
To fix the problem, this commit adds an `init` function to `fix_stacks.py` so
that `fix-stacks` can be initialized eagerly, hopefully before any file
descriptors for files are open.
For `dmd.py`, other than fixing the AWSY problems, this has little effect,
because `fix-stacks` is always initialized.
For `utils.py`, which is used to process the output of most tests, this has a
more noticeable effect: the `fix-stacks` process is always spawned, rather than
being spawned only when needed. If no stack traces appear in the test output,
this means that `fix-stacks` is spawned unnecessarily. But it's cheap to spawn;
the expensive part only happens when stack traces start getting fixed. So I
think this change in behaviour is acceptable.
The commit also renames some things to more standard Python style, e.g.
`json_mode` instead of `jsonMode`.
Differential Revision: https://phabricator.services.mozilla.com/D69478
--HG--
extra : moz-landing-system : lando
N_STACK_BYTES are needed to store backtrace information, so use
that plus some extra as the minimum stack size for the sampler
thread to ensure that it doesn't overflow.
Differential Revision: https://phabricator.services.mozilla.com/D65705
--HG--
extra : moz-landing-system : lando
This prevents a race condition where the watchman hook can potentially
invalidate the cache in-between the call to 'isfile' and 'getmtime'.
Depends on D69189
Differential Revision: https://phabricator.services.mozilla.com/D69190
--HG--
extra : moz-landing-system : lando
The bugbug scheduler currently chooses which manifests are important, and we
then run *every* task that contains those manifests. This is likely overkill
and we can reduce the number of configurations we run these manifests on.
Differential Revision: https://phabricator.services.mozilla.com/D68466
--HG--
extra : moz-landing-system : lando
This ensures we fail with invalid module paths early. Otherwise users wouldn't
find out until the decision task fails.
Differential Revision: https://phabricator.services.mozilla.com/D68464
--HG--
extra : moz-landing-system : lando
These routes are applied to all tasks except the decision task. There
is currently no validity checking in the frontend, so if the provided
routes aren't valid the decision task will fail.
Differential Revision: https://phabricator.services.mozilla.com/D67828
--HG--
extra : moz-landing-system : lando
This adds an optional routes key to the task_task_config schema,
which is a list of strings. Anything in this key is added to the list
of routes for tasks scheduled by the decision task.
Differential Revision: https://phabricator.services.mozilla.com/D67827
--HG--
extra : moz-landing-system : lando
The bugbug scheduler currently chooses which manifests are important, and we
then run *every* task that contains those manifests. This is likely overkill
and we can reduce the number of configurations we run these manifests on.
Differential Revision: https://phabricator.services.mozilla.com/D68466
--HG--
extra : moz-landing-system : lando
This ensures we fail with invalid module paths early. Otherwise users wouldn't
find out until the decision task fails.
Differential Revision: https://phabricator.services.mozilla.com/D68464
--HG--
extra : moz-landing-system : lando
This also upgrades the vendored Glean version, which thanks to our
upstream work doesn't change anything else (except one small
Mozilla-developed dependency)
It's still feature-gated to nightly.
In C++ there's now a `MOZ_GLEAN` define.
For Rust it's behind the `glean` feature (enabled on nightly only).
The `fog` crate is empty, so no Glean is actually instantiated.
Differential Revision: https://phabricator.services.mozilla.com/D68539
--HG--
rename : third_party/rust/glean-preview/src/metrics/mod.rs => toolkit/components/glean/src/lib.rs
extra : moz-landing-system : lando
This also upgrades the vendored Glean version, which thanks to our
upstream work doesn't change anything else (except one small
Mozilla-developed dependency)
It's still feature-gated to nightly.
In C++ there's now a `MOZ_GLEAN` define.
For Rust it's behind the `glean` feature (enabled on nightly only).
The `fog` crate is empty, so no Glean is actually instantiated.
Differential Revision: https://phabricator.services.mozilla.com/D68539
--HG--
rename : third_party/rust/glean-preview/src/metrics/mod.rs => toolkit/components/glean/src/lib.rs
extra : moz-landing-system : lando
Also adds missing includes in some files, these were previously only transivitely
included through mozilla/TypeTraits.h.
Differential Revision: https://phabricator.services.mozilla.com/D68561
--HG--
extra : moz-landing-system : lando
We disable this checker until Synopsis fixes this or provides a valid workaround.
Differential Revision: https://phabricator.services.mozilla.com/D68429
--HG--
extra : moz-landing-system : lando
Also update the update-rust.sh script in a couple ways:
- Stop copying the source for mp4parse_fallible into the tree; use crates.io
- Include submodules in mp4parse-rust checkout; needed for tests
- Exclude unnecessary build.rs from mp4parse_fallible, it was causing problems
- Update mp4rust_capi/Cargo.toml patch to exclude cdylib. It's only necessary for test_ffi and causes build problems otherwise
Differential Revision: https://phabricator.services.mozilla.com/D68139
--HG--
extra : moz-landing-system : lando
I'm unsure of the root cause as the file should exist if generate_tasks has worked, but this should avoid errors in the meantime while we investigate.
Differential Revision: https://phabricator.services.mozilla.com/D67833
--HG--
extra : moz-landing-system : lando
We still need it for Android < 6.0 (API 23) because otherwise we don't
have a way to hook up mozalloc.
Differential Revision: https://phabricator.services.mozilla.com/D66993
--HG--
extra : moz-landing-system : lando
There appears to be a bug in babel-eslint where it still uses the old
`ExperimentalSpreadProperty` and `ExperimentalRestProperty` node types when
babel itself (and hence the AST visitor keys) have switched to `SpreadElement`
and `RestElement`. I suspect there isn't much point in filing a babel-eslint
issue for this, they are currently in the process of working on v11 which does
fix the issue but is currently only available as a beta release.
There are three alternatives:
1. Use the beta babel-eslint.
2. Hack the visitor keys to include the old node types.
3. Traverse the AST changing the nodes from the old types to the new types.
Using a beta version is probably not a great idea so this patch does 2 which is
the simplest of the other options. It does have the drawback however that if
any eslint rules we use rely on the new node type names they will fail to work.
Differential Revision: https://phabricator.services.mozilla.com/D66998
--HG--
extra : moz-landing-system : lando
When a package is installed with "pip --user", the import works right after the virtualenv activation.
Even if we force its install via mozbuild virtualenv it will fail. This patch checks if the installed package is **inside** or
**outside** the virtualenv. If it's outside, it forces its installation **inside**.
Notice that we might want to move that logic in mozbuild/virtualenv.py
Differential Revision: https://phabricator.services.mozilla.com/D67592
--HG--
extra : moz-landing-system : lando
In the failure case, errors has value, but output doesn't. So previously we were printing the errors but returning success to the lint handler.
Differential Revision: https://phabricator.services.mozilla.com/D67670
--HG--
extra : moz-landing-system : lando
Adds --config-paths parameter to mach mozlint where test configuration
files (.yml) can be located. The default path "tools/lint" is set automatically
and additional paths have preference over the default.
This allows Thunderbird to run mozlint-based tests in Taskcluster with a
different configuration.
Differential Revision: https://phabricator.services.mozilla.com/D67077
--HG--
extra : moz-landing-system : lando
We still need it for Android < 6.0 (API 23) because otherwise we don't
have a way to hook up mozalloc.
Differential Revision: https://phabricator.services.mozilla.com/D66993
--HG--
extra : moz-landing-system : lando
This includes several fixes required to build against musl libc.
Conflicts were resolved in 00-arm-exidx-rollup.patch and
10-json-upload.patch. 08-dont-add-sp-to-clobber-list.patch was
applied upstream and is no longer needed. The others applied cleanly.
breakpad_getcontext.S is now built conditionally based upon the
available of getcontext() from libc, rather than only on Android.
The profiler was updated to reflect this change.
Differential Revision: https://phabricator.services.mozilla.com/D67108
--HG--
rename : toolkit/crashreporter/breakpad-patches/09-gnu-alt-form-minimal-support.patch => toolkit/crashreporter/breakpad-patches/08-gnu-alt-form-minimal-support.patch
rename : toolkit/crashreporter/breakpad-patches/10-json-upload.patch => toolkit/crashreporter/breakpad-patches/09-json-upload.patch
rename : toolkit/crashreporter/google-breakpad/src/common/android/breakpad_getcontext.S => toolkit/crashreporter/google-breakpad/src/common/linux/breakpad_getcontext.S
rename : toolkit/crashreporter/google-breakpad/src/common/android/breakpad_getcontext_unittest.cc => toolkit/crashreporter/google-breakpad/src/common/linux/breakpad_getcontext_unittest.cc
extra : moz-landing-system : lando
Add generated files to "Generated.txt" to exclude them from being linted to
avoid a "trailing whitespace" error in a comment copied over from the original
Unicode files.
Depends on D66557
Differential Revision: https://phabricator.services.mozilla.com/D66558
--HG--
extra : moz-landing-system : lando
Stop including "Linker.h" and always include <link.h> instead, which also comes with ElfW() and other things that this code needs.
Also fix up "!/" path detection code.
Differential Revision: https://phabricator.services.mozilla.com/D64997
--HG--
extra : moz-landing-system : lando
This removes most dependencies on BlocksRingBuffer, to ease the transition to
the upcoming Fission-friendly profile buffer, including:
- Length type,
- SumBytes(),
- Gecko extensions of serialization.
Differential Revision: https://phabricator.services.mozilla.com/D66722
--HG--
rename : tools/profiler/public/BlocksRingBufferGeckoExtensions.h => tools/profiler/public/ProfileBufferEntrySerializationGeckoExtensions.h
extra : moz-landing-system : lando
This uses the `email` module to parse the .patch file that Github generates,
so that it properly decodes encoded-words in the headers. Also using this
module is better with python3, so this patch also takes the command off the
python2 whitelist and makes it python3-compatible.
Differential Revision: https://phabricator.services.mozilla.com/D66621
--HG--
extra : moz-landing-system : lando
The babel-eslint parser defaults to assuming all scripts are modules. Although
we're moving in that direction it seems reasonable to set the default as regular
scripts for now. All the places that were previously overriding the parser are
already specifying the sourceType in their eslint configs.
I chose to put the babel config in a file that babel itself won't look to avoid
the risk of this config impacting the use of babel elsewhere in the tree.
Differential Revision: https://phabricator.services.mozilla.com/D66127
--HG--
extra : moz-landing-system : lando
Neither of these selectors involve the user choosing tasks, so showing
estimates or saving them in the 'mach try again' history doesn't make much
sense.
Differential Revision: https://phabricator.services.mozilla.com/D66628
--HG--
extra : moz-landing-system : lando
This allows us to change the default optimization strategy used in try pushes.
While probably not super useful to developers, it can help us easily test
changes to new and experimental optimizations on try.
This also changes the default to the 'bugbug_push_schedules' strategy, since
SETA is more or less random and shouldn't be used by 'mach try auto'. In the
future, we'll switch this back to simply using the default optimization as the
default will ideally be the best one that we have.
Differential Revision: https://phabricator.services.mozilla.com/D65746
--HG--
extra : moz-landing-system : lando
The 'auto' in 'mach try auto' stands for two things:
1. It automatically picks tasks for you.
2. It runs the same scheduling algorithms as autoland.
It accomplishes this by creating a new target_tasks method that spoofs the
'project' parameter to autoland.
Differential Revision: https://phabricator.services.mozilla.com/D60184
--HG--
extra : moz-landing-system : lando
Now that `fix_stacks.py` is being used instead of `fix_macosx_stack.py`,
stack-fixing time has dropped from about 14 minutes to about 30 seconds on my
new MacBook Pro.
Also, print a warning about stacks not being fixed if `MOZ_DISABLE_STACK_FIX`
is set. This warning shows up at the start of the test run.
Also, print a warning about stack fixing slowness, because 30 seconds is long
enough to possibly be surprising. This warning shows up just before the first
stack frame is fixed, like this:
```
Assertion failure: false (BEEP BOOP), at /home/njn/moz/au3/dom/base/nsGlobalWindowOuter.cpp:1342
Initializing stack-fixing for the first stack frame, this may take a while...
#01: nsGlobalWindowOuter::nsGlobalWindowOuter(unsigned long) (/home/njn/moz/au3/dom/base/nsGlobalWindowOuter.cpp:1342)
#02: ...
```
Differential Revision: https://phabricator.services.mozilla.com/D65676
--HG--
extra : moz-landing-system : lando
When we are running from a network drive the new feature in part 1 doesn't work.
So this uses DuplicateHandle instead of OpenThread to get the thread handle used
by the profiler.
It also removes a DuplicateHandle THREAD_ALL_ACCESS call that also fails and a
DuplicateHandle to get a real process handle, which only seems to have been to
fix something on Windows XP.
The handle passed in is always the profiler one, so already has the necessary
permissions. If no thread handle is passed then the pseudo handle is used.
Differential Revision: https://phabricator.services.mozilla.com/D66611
--HG--
extra : moz-landing-system : lando
When we are running from a network drive the new feature in part 1 doesn't work.
So this uses DuplicateHandle instead of OpenThread to get the thread handle used
by the profiler.
It also removes a DuplicateHandle THREAD_ALL_ACCESS call that also fails and a
DuplicateHandle to get a real process handle, which only seems to have been to
fix something on Windows XP.
The handle passed in is always the profiler one, so already has the necessary
permissions. If no thread handle is passed then the pseudo handle is used.
Depends on D66610
Differential Revision: https://phabricator.services.mozilla.com/D66611
--HG--
extra : moz-landing-system : lando
Add filename support for plain display
Update fzf version in tests
Update tooltool url for fzf
Differential Revision: https://phabricator.services.mozilla.com/D66571
--HG--
extra : moz-landing-system : lando
N_STACK_BYTES are needed to store backtrace information, so use
that plus some extra as the minimum stack size for the sampler
thread to ensure that it doesn't overflow.
Differential Revision: https://phabricator.services.mozilla.com/D65705
--HG--
extra : moz-landing-system : lando
The new ProfileBufferEntryReader/Writer are now used everywhere, including in
the profilers and tests.
The old EntryReader/Writer have been removed.
Differential Revision: https://phabricator.services.mozilla.com/D65697
--HG--
extra : moz-landing-system : lando
Specifically:
- handle input that doesn't end in a newline;
- only pass lines that match the regexp to `fix-stacks`.
These changes are needed for `utils.py` to use `fix_stacks.py` without
problems.
Differential Revision: https://phabricator.services.mozilla.com/D65944
--HG--
extra : moz-landing-system : lando
Remove use of requests module in preview pane
Reformat task duration data to avoid reprocessing in preview pane
Avoid loading task durations json more than once.
Increase required fzf version, use temporary file instead of arglist
Differential Revision: https://phabricator.services.mozilla.com/D65094
--HG--
extra : moz-landing-system : lando
N_STACK_BYTES are needed to store backtrace information, so use
that plus some extra as the minimum stack size for the sampler
thread to ensure that it doesn't overflow.
Differential Revision: https://phabricator.services.mozilla.com/D65705
--HG--
extra : moz-landing-system : lando
This removes the obsolete backend. Notes on some of the less obvious changes
made as part of this patch:
- some of the gFoo style getters in Blocklist.jsm were only used by the XML
version of the blocklist; I've removed them and tried to remove spurious
settings of those properties in the remaining tests.
- some utility methods (e.g. distribution information getters) were also only
used for the XML version (for the update URL).
- it's no longer necessary to test switching implementations.
- in browser/base/content/test/plugins/, we ran some tests from two manifests
in order to run them with both blocklist backends. The simplest way of
reducing this back down to one was to remove the remote-settings one. If I'd
been more future-oriented when I created the duplication, perhaps I would
have moved the XML version out into a different manifest instead, but I
didn't, so now it looks like we're removing the modern one, whereas really
we're going to be running the modern one as part of the "normal" tests and
we're no longer running the "old" tests.
- removed all mentions I could see of extensions.blocklist.url which is no
longer used for anything.
- per https://bugzilla.mozilla.org/show_bug.cgi?id=1016555#c23, updated
references for the OneCRL timing and how it relates to blocklist updates.
Differential Revision: https://phabricator.services.mozilla.com/D64933
--HG--
extra : moz-landing-system : lando
N_STACK_BYTES are needed to store backtrace information, so use
that plus some extra as the minimum stack size for the sampler
thread to ensure that it doesn't overflow.
Differential Revision: https://phabricator.services.mozilla.com/D65705
--HG--
extra : moz-landing-system : lando
There is a logic error in the code that is not catching missing tests. Also, the manifest path for each test needs to be added along with the test name so the logger can be able to print it.
Differential Revision: https://phabricator.services.mozilla.com/D65473
--HG--
extra : moz-landing-system : lando
On Linux and Mac, this makes `dmd.py` *much* faster when it is first run on a
DMD data file.
On Windows, this makes DMD actually usable locally. Previously the stacks
weren't fixed and so were rubbish.
Differential Revision: https://phabricator.services.mozilla.com/D57271
--HG--
extra : moz-landing-system : lando
Changes:
Remove `ubuntu-bionic` flag that was used during development to enable use of ubuntu1804-test docker image.
Remove unnecessary conditional and in the process rewrite how `runtests.py` checks the environment for `pactl` prior to running mochitests.
Differential Revision: https://phabricator.services.mozilla.com/D65389
--HG--
extra : moz-landing-system : lando
Currently we only profile the Java Main Thread, and don't profile anything
else. This is not ideal, but this is how it works right now. And inside the
code index `0` was hardcoded on the most parts of the code. We can rollback
this patch once we want to implement profiling more than one thread, or we can
think about something more clever.
Differential Revision: https://phabricator.services.mozilla.com/D64752
--HG--
extra : moz-landing-system : lando
CLOSED TREE
Backed out changeset 01548614184b (bug 1602773)
Backed out changeset 430c8e6b0c5a (bug 1602773)
Backed out changeset 1b4e2b044fcd (bug 1602773)
To facilitate the upcoming transition to a new Fission-friendly storage, all
uses of `BlocksRingBuffer::{,Block}Index` are replaced with
`ProfileBuffer{,Block}Index`.
`BlocksRingBuffer::{,Block}Index` are not needed anymore.
Differential Revision: https://phabricator.services.mozilla.com/D64516
--HG--
extra : moz-landing-system : lando
In |mach try fuzzy| there's a TARGET_TASK_FILTERS variable that we use to make
it more difficult to run certain kinds of expensive / resource constrained
tasks. The problem is that the only way to run these is to use '--full' at
which point there's no way to distinguish which tasks are "valid" (i.e run on
mozilla-central) and which tasks don't.
This adds a flag that will ensure the default set truly matches what we run
on central.
Differential Revision: https://phabricator.services.mozilla.com/D64648
--HG--
extra : moz-landing-system : lando
Toshihito Kikuchi
Gerald Squelart
Dzmitry Malyshau
Bogdan Tara
Sylvestre Ledru
Markus Stange
Kershaw Chang
Butkovits Atila
Nicholas Nethercote
Myeongjun Go
Greg Tatum
Tilden Windsor
Tom Prince
championshuttler
Coroiu Cristina
Marco Castelluccio
Kris Maglione
alwu
Aaron Klotz
Simon Fraser
Edouard Oger
Tarek Ziadé
Mitchell Hentges
Kartikaya Gupta
Nazım Can Altınova
Christian Holler
Mihai Alexandru Michis
Jon Bauman
Chris Fronk
thomasmo
Geoff Brown
Jonathan Watt
Simon Giesecke
Cosmin Sabou
Greg V
Ian Moody
Julien Wajsberg
undef1nd
Andrew Halberstadt
Michael Hoffmann
Matthew Gaudet
David Teller
Stefan Hindli
Bob Clary
Csoregi Natalia
Joel Maher
Emilio Cobos Álvarez
Mike Hommey
Arthur Iakab
Simon Sapin
Edwin Takahashi
Gijs Kruitbosch
Tooru Fujisawa
Ciure Andrei
Mark Banner
Jesse Schwartzentruber
Anurag Aggarwal
Andi-Bogdan Postelnicu
Chris Peterson
Ricky Stewart
Neil Deakin
MahakBansal
Andreas Farre
Florian Quèze
Michael Forney
Eric Rahm
Narcis Beleuzu
Andrew Swan
James Graham
Jan-Erik Rediger
Razvan Maries
André Bargull
Gregory Mierzwinski
byron jones
James Willcox
Dave Townsend
Rob Lemley
Ed Lee
William Lachance
Julian Descottes
Zibi Braniecki
Bob Owen
Daniel Varga
alexandru.ionescu
Dorel Luca
Alessio Placitelli