Check `profiler_is_locked_on_current_thread()` before recording an IPC marker.
This removes the deadlock found in bug 1675406:
- SamplerThread: During sampling, some data is recorded into the profile buffer, which locks ProfileChunkedBuffer::mMutex, this triggers some chunk updates that are sent to ProfileBufferGlobalController, which attempts to lock its mutex.
- Main thread: An IPC with an update arrives, ProfileBufferGlobalController locks its mutex, then it sends an IPC out, this records a marker into ProfileChunkedBuffer, which attempts to lock its mMutex.
With this patch and bug 1671403, that last IPC will not record a marker anymore.
Differential Revision: https://phabricator.services.mozilla.com/D96971
When running as a "universal" build, use an x64 GMP child process if the CDM library is an x64 binary.
Use ifdefs extensively to reduce risk to Intel builds if the fix needs to be uplifted.
Requires a server-side balrog change to serve an Intel Widevine binary to ARM browser versions.
Differential Revision: https://phabricator.services.mozilla.com/D96288
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
The support code for Android, which doesn't support shm_open and can't
use the memfd backend because of issues with its SELinux policy (see bug
1670277), has been reorganized to reflect that we'll always use its own
API, ashmem, in that case.
Differential Revision: https://phabricator.services.mozilla.com/D90605
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
The support code for Android, which doesn't support shm_open and can't
use the memfd backend because of issues with its SELinux policy (see bug
1670277), has been reorganized to reflect that we'll always use its own
API, ashmem, in that case.
Differential Revision: https://phabricator.services.mozilla.com/D90605
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
Differential Revision: https://phabricator.services.mozilla.com/D90605
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
Differential Revision: https://phabricator.services.mozilla.com/D90605
Some Android ARM64 devices appear to have a bug where sendmsg sometimes
returns 0xFFFFFFFF, which we're assuming is a -1 that was incorrectly
truncated to 32-bit and then zero-extended. This patch detects that
value (which should never legitimately be returned, because it's 16x
the maximum message size) and replaces it with -1, with some additional
assertions.
The workaround is also enabled on x86_64 Android on debug builds only,
so that the code has CI coverage.
Differential Revision: https://phabricator.services.mozilla.com/D89845
posix_fallocate iterates over each page/block in a shmem to ensure the
OS allocates memory to back it. Large shmems will cause many read/write
calls to be made, and when profiling, it is very likely a SIGPROF signal
will interrupt us at sufficiently high sampling rates. Most attempts at
retrying will fail for the same reason, and this can cause the threads
to block for an indeterminate period of time.
To work around this we use the profiler's "thread sleep" mechanism to
indicate that the sampler thread should not interrupt this thread with
the sampling signal more than once.
Differential Revision: https://phabricator.services.mozilla.com/D87373
The goal of this patch is to reduce memory usage. On at least OSX, std::Queue
can use 4kb of memory even with nothing in it. This can be around 52kb of
memory per content process.
The segment size of 64 is fairly arbitrary, but these queues didn't have
more than a few hundred items in them, so it seemed like a reasonable
trade off between space for mostly-empty queues and segment overhead.
Differential Revision: https://phabricator.services.mozilla.com/D87325
An actual use of the `BACKGROUND_X11` thread may have briefly existed in
2009 but probably never shipped; it's been unused since then, and even
the comments mentioning it have been pruned (bug 909028, bug 624422).
The other thread types besides IO have been commented out ever since
they were first committed.
This patch also removes `x11_util.h`, which has been unused since 2017
(bug 1426284); earlier in history, it had one of those comments
mentioning the nonexistent X11 thread.
Differential Revision: https://phabricator.services.mozilla.com/D85346
The Chromium-derived IPC code was, as far as I can tell, originally
designed for Windows and assumed that channels would be named pipes,
managed and connected to via `std::wstring` paths. The port to Unix,
however, used unnamed `socketpair()` and passed them directly from
process to process, so it has no use for these channel IDs... but it
still computes and propagates them, even though they're not used, using
deprecated wide-string APIs.
This patch introduces a typedef for an abstract channel ID, which is a
`wstring` on Windows and an empty struct on Unix, to allow removing the
string code where it's not needed without needing to completely redesign
the channel abstraction.
Differential Revision: https://phabricator.services.mozilla.com/D72260
Chromium's fix for CVE-2011-3079 added an optional prefix parameter for
channel IDs, but we've never used it and have no plans to. (Chromium
itself doesn't appear to have used it except with the prefixes "gpu"
and "nacl", and the code has since been removed completely in favor of
Mojo.) So let's simplify things and remove it.
Differential Revision: https://phabricator.services.mozilla.com/D84276
This "create a pipe" operation has a mode where, on Unix, it doesn't
create a new transport but rather uses a hard-coded fd for the initial
IPC channel in a child process. (It was originally written for Windows
and the assumption of using named pipes and pathnames for everything.)
That seems like a footgun, so this patch checks for trying to "create"
that pipe twice. However, it doesn't check for accidentally calling it
in the parent process.
Differential Revision: https://phabricator.services.mozilla.com/D72259
The PipeMap class tries to simulate the Windows channel model (named
pipes that the client opens by a pathname) on Unix. However, it's
effectively dead code -- the map is empty except in some unit tests that
we never imported.
What we do is generate a "channel ID" with string formatting, then don't
pass it to the child or ever insert anything into the map, then the child
looks up an empty string and doesn't find it, so it uses the hard-coded
fixed fd for the initial channel.
Basically, it does nothing except maybe confuse unfamiliar readers, so
let's get rid of it.
Differential Revision: https://phabricator.services.mozilla.com/D72258
WebRender makes extensive use of shared memory buffers, particularly for
images decoded in the content process. These images can be arbitrarily
large, and there being insufficient memory for an allocation must be
handled gracefully.
On Linux, we will currently crash with a SIGBUS signal during image
decoding instead of just displaying the broken image tag. This is
because the pages backing the shared memory are only allocated when we
write to them. This blocks shipping WebRender on Linux.
This patch uses posix_fallocate to force the reservation of the pages,
and allows failing gracefully if they are unavailable.
Differential Revision: https://phabricator.services.mozilla.com/D80650
CVE-2018-4435 (https://crbug.com/project-zero/1671) was fixed in macOS
10.12 and up, but when we added uses of shm_open that would be affected
by it we still supported 10.9, so we added a workaround that tests for
the bug (by trying to exploit it) and falls back to the slower
alternative of temporary files if necessary.
The minimum supported version is now 10.12, and we've already
committed changes (e.g., to sandboxing) that would break the browser
on older versions, so we can remove this code. Note that we also have
cross-platform gtests that check for this type of bug, so we'll have some
warning if it's ever reintroduced.
Differential Revision: https://phabricator.services.mozilla.com/D83197
We'll never read more than MAX_DESCRIPTORS_PER_MESSAGE file descriptors in a
single message, so size the buffer based on that value.
Differential Revision: https://phabricator.services.mozilla.com/D79162
This adds 3 new profiler markers for each IPC message:
* One just before the first byte is sent over the IPC channel
* One just after the last byte is sent over the IPC channel
* One just after the last byte is received from the IPC channel
With the already-existing IPC markers (for when SendXXX and RecvXXX are
called), this allows us to calculate the following statistics:
* Send thread latency
* IPC send duration
* IPC recv latency
* Recv thread latency
For more information on how this is presented in the UI, see:
<https://github.com/firefox-devtools/profiler/pull/2535>.
Differential Revision: https://phabricator.services.mozilla.com/D70790
We want it to returning the actual nsThread if that's where the MessageLoop would dispatch its tasks; otherwise return the MessageLoop's EventTarget
Depends on D80357
Differential Revision: https://phabricator.services.mozilla.com/D80811
Florian Quèze
Sylvestre Ledru
Markus Stange
Simon Giesecke
Gerald Squelart
Razvan Maries
Mihai Alexandru Michis
Haik Aftandilian
Ricky Stewart
Bogdan Tara
Jed Davis
Dorel Luca
Bas Schouten
shawnjohnjr
Andrew Osmond
Andrew McCreight
Butkovits Atila
Kashav Madan
Jim Porter
Jean-Yves Avenard