Removes the PPluginSurface actor used for windowed plugins, as part of removing all of NPAPI plugin support. SharedDIB is then unused and is also removed.
Differential Revision: https://phabricator.services.mozilla.com/D107140
`High` priority is being used for vsync tasks, so we should rename it to
make it clear, and renaming it also makes our priority naming less
confusing.
Differential Revision: https://phabricator.services.mozilla.com/D109536
Removes the PPluginSurface actor used for windowed plugins, as part of removing all of NPAPI plugin support. SharedDIB is then unused and is also removed.
Differential Revision: https://phabricator.services.mozilla.com/D107140
When setting up calls to `sendmsg` for IPC on Unix systems, we generate
`iovec`s for the entire message or until the `IOV_MAX` limit is reached,
whichever comes first. However, messages can be very large (up to 256
MiB currently), while the OS socket buffer is relatively small (8KiB on
macOS and FreeBSD, ~200KiB on Linux).
This patch detects the socket buffer size with the `SO_SNDBUF` socket
option and cuts off the `iovec` array after it's reached; it also adjusts
the Linux sandbox policy to allow reading that value in all processes.
On my test machines this increases throughput on large messages by about
2.5x on macOS (from ~0.3 to ~0.7 GB/s), but on Linux the improvement is
only about 5% (most of the running time is spent elsewhere).
Differential Revision: https://phabricator.services.mozilla.com/D105852
Currently we walk through the entire list of not-yet-written IPC buffers
when building the gathered I/O list for `sendmsg`, to determine the
total remaining length of the messages, even after reaching the OS's
limit on how many `iovec`s it will accept in one call.
This patch halts the iteration when we reach the `iovec` limit, because
we don't need the exact length; it's sufficient to know whether the
entire message was written, which is impossible in that case.
This increases throughput on large messages by about 7x on macOS (from
~0.04 to ~0.3 GB/s) and 1.7x on Linux (from ~0.3 to ~0.5 GB/s), on my
test machines. The effect is more significant on macOS because its
smaller socket buffer size (8kB vs. ~200kB) means we spend more time
setting up the syscall per unit data copied; see also the next patch.
Differential Revision: https://phabricator.services.mozilla.com/D105851
This point in the code is the end of receiving a message, not the start
(note the `if (partial) { break; }` above), so it should be marked
accordingly. The profiler frontend is expecting the end marker;
currently two of the four reported time intervals are `unknown` on
Windows, and this patch fixes that.
(Recording the receiving start time is complicated, because we don't
have a `Messsage` object until we've read the buffer with the (end of
the) header, and it might make more sense to timestamp it before the
first receive operation. Currently, neither channel implementation
attempts this.)
Differential Revision: https://phabricator.services.mozilla.com/D105094
Currently, we add an `eSending`/`TransferStart` marker every time we
send data as part of a message, rather than only the first time. The
profiler frontend appears to ignore the extra markers (the displayed time
intervals look reasonable), but it's wasteful: on large messages it can
consume enough CPU time to appear in the profile itself.
Differential Revision: https://phabricator.services.mozilla.com/D105093
There are no code changes, only #include changes.
It was a fairly mechanical process: Search for all "AUTO_PROFILER_LABEL", and in each file, if only labels are used, convert "GeckoProfiler.h" into "ProfilerLabels.h" (or just add that last one where needed).
In some files, there were also some marker calls but no other profiler-related calls, in these cases "GeckoProfiler.h" was replaced with both "ProfilerLabels.h" and "ProfilerMarkers.h", which still helps in reducing the use of the all-encompassing "GeckoProfiler.h".
Differential Revision: https://phabricator.services.mozilla.com/D104588
Check `profiler_is_locked_on_current_thread()` before recording an IPC marker.
This removes the deadlock found in bug 1675406:
- SamplerThread: During sampling, some data is recorded into the profile buffer, which locks ProfileChunkedBuffer::mMutex, this triggers some chunk updates that are sent to ProfileBufferGlobalController, which attempts to lock its mutex.
- Main thread: An IPC with an update arrives, ProfileBufferGlobalController locks its mutex, then it sends an IPC out, this records a marker into ProfileChunkedBuffer, which attempts to lock its mMutex.
With this patch and bug 1671403, that last IPC will not record a marker anymore.
Differential Revision: https://phabricator.services.mozilla.com/D96971
When running as a "universal" build, use an x64 GMP child process if the CDM library is an x64 binary.
Use ifdefs extensively to reduce risk to Intel builds if the fix needs to be uplifted.
Requires a server-side balrog change to serve an Intel Widevine binary to ARM browser versions.
Differential Revision: https://phabricator.services.mozilla.com/D96288
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
The support code for Android, which doesn't support shm_open and can't
use the memfd backend because of issues with its SELinux policy (see bug
1670277), has been reorganized to reflect that we'll always use its own
API, ashmem, in that case.
Differential Revision: https://phabricator.services.mozilla.com/D90605
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
The support code for Android, which doesn't support shm_open and can't
use the memfd backend because of issues with its SELinux policy (see bug
1670277), has been reorganized to reflect that we'll always use its own
API, ashmem, in that case.
Differential Revision: https://phabricator.services.mozilla.com/D90605
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
Differential Revision: https://phabricator.services.mozilla.com/D90605
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
Differential Revision: https://phabricator.services.mozilla.com/D90605
Some Android ARM64 devices appear to have a bug where sendmsg sometimes
returns 0xFFFFFFFF, which we're assuming is a -1 that was incorrectly
truncated to 32-bit and then zero-extended. This patch detects that
value (which should never legitimately be returned, because it's 16x
the maximum message size) and replaces it with -1, with some additional
assertions.
The workaround is also enabled on x86_64 Android on debug builds only,
so that the code has CI coverage.
Differential Revision: https://phabricator.services.mozilla.com/D89845
posix_fallocate iterates over each page/block in a shmem to ensure the
OS allocates memory to back it. Large shmems will cause many read/write
calls to be made, and when profiling, it is very likely a SIGPROF signal
will interrupt us at sufficiently high sampling rates. Most attempts at
retrying will fail for the same reason, and this can cause the threads
to block for an indeterminate period of time.
To work around this we use the profiler's "thread sleep" mechanism to
indicate that the sampler thread should not interrupt this thread with
the sampling signal more than once.
Differential Revision: https://phabricator.services.mozilla.com/D87373
The goal of this patch is to reduce memory usage. On at least OSX, std::Queue
can use 4kb of memory even with nothing in it. This can be around 52kb of
memory per content process.
The segment size of 64 is fairly arbitrary, but these queues didn't have
more than a few hundred items in them, so it seemed like a reasonable
trade off between space for mostly-empty queues and segment overhead.
Differential Revision: https://phabricator.services.mozilla.com/D87325
David Parks
Brindusan Cristian
Sean Feng
Csoregi Natalia
Simon Giesecke
Florian Quèze
Jed Davis
Gerald Squelart
Sylvestre Ledru
Markus Stange
Razvan Maries
Mihai Alexandru Michis
Haik Aftandilian
Ricky Stewart
Bogdan Tara
Dorel Luca
Bas Schouten
shawnjohnjr
Andrew Osmond
Andrew McCreight