This moves most of the JsepTrack functionality for dealing with duplicate
payload types to a new EnsurePayloadTypeNotDuplicate method in
JsepCodecDescription. It also adds a virtual EnsureNoDuplicatePayloadTypes
method that checks the appropriate payload types for each codec type.
Differential Revision: https://phabricator.services.mozilla.com/D92112
There is a case where two VideoOutputs are feeding the same VideoFrameContainer
with data: VideoStreamTrack::AddVideoOutput(VideoFrameContainer) and
VideoStreamTrack::AddVideoOutput(FirstFrameVideoOutput) in HTMLMediaElement.
This is racy since they both call VideoFrameContainer::NewFrameID().
These VideoOutputs use different ProducerIDs however, so it makes sense for them
to also use separate FrameID generators.
Differential Revision: https://phabricator.services.mozilla.com/D94054
The purpose for those reftest at present isn't to verify that our image conversion is correct (it isn't), just that decoding produce something not completely broken
Differential Revision: https://phabricator.services.mozilla.com/D94065
We have reports of VPx WMF failures in the wild causing performance issues, but
it can be hard to detect these failures. Having explicit markers in the profiler
with more information will make it easier to see if users are running into these
issues if they can provide profiles. I've also added logs to enrich cases where
we will use logging to debug.
Driveby fix a comment that says we're asserting when we no longer are.
Differential Revision: https://phabricator.services.mozilla.com/D93672
We have reports of VPx WMF failures in the wild causing performance issues, but
it can be hard to detect these failures. Having explicit markers in the profiler
with more information will make it easier to see if users are running into these
issues if they can provide profiles. I've also added logs to enrich cases where
we will use logging to debug.
Driveby fix a comment that says we're asserting when we no longer are.
Differential Revision: https://phabricator.services.mozilla.com/D93672
We have reports of VPx WMF failures in the wild causing performance issues, but
it can be hard to detect these failures. Having explicit markers in the profiler
with more information will make it easier to see if users are running into these
issues if they can provide profiles. I've also added logs to enrich cases where
we will use logging to debug.
Driveby fix a comment that says we're asserting when we no longer are.
Differential Revision: https://phabricator.services.mozilla.com/D93672
It would be good for us to know how many websites actually use MediaSession API to control (play/pause/stop) their media playback, an how many websites use our default handler.
Differential Revision: https://phabricator.services.mozilla.com/D93283
Each platform has different ways to allow users to use media control, adding a telemetry probe to detect that would be good for us to know the usage among different platforms.
Differential Revision: https://phabricator.services.mozilla.com/D93282
It would be good for us to know how many websites actually use MediaSession API to control (play/pause/stop) their media playback, an how many websites use our default handler.
Differential Revision: https://phabricator.services.mozilla.com/D93283
Each platform has different ways to allow users to use media control, adding a telemetry probe to detect that would be good for us to know the usage among different platforms.
Differential Revision: https://phabricator.services.mozilla.com/D93282
This interface is never used directly, and the only consumers of the virtual functions are by the derived classes themselves.
Differential Revision: https://phabricator.services.mozilla.com/D93180
Since the semantics of `ContentParent::MarkAsDead` are significantly different
from `GeckoProcessManager::MarkAsDead`, let's rename the latter to better
reflect what it actually does.
Differential Revision: https://phabricator.services.mozilla.com/D92649
- What this patch does:
Send a "download-suspended-by-cache" signal to the media-element paired
with a cloned cache-stream right after the cache-stream is cloned. That
signal help the media-element to decide if it's ready to enter the
HAVE_ENOUGH_DATA state and fire a "canplaythrough" event.
- Why this patch is needed:
It solves a WPT timeout issue. That WPT test waits a "canplaythrough"
event.
- What problem this patch solves is:
This patch addresses the problem mentioned in [1].
Each media-element pairs with a cache-stream downloading the data it
needs. When the data-download of a cache-stream gets suspended, the
media-element paired with the cache-stream should be notified. This
notification is one of the factor to move the ready-state of the
media-element to HAVE_ENOUGH_DATA. However, the media-element paired
with a cloned cache-stream may never receive this notification. And the
worst is that it may never have a chance to download enough data it
needs to move the ready-state to HAVE_ENOUGH_DATA as well at the same
time, in the case mentioned below.
This can happen when a media-element paired with a cloned cache-stream
is created after the cache is full, and all of the cache-streams are
suspended. (Cloned cache-stream is a cache-stream cloned from another
cache-stream that shares the same underlying data with it since their
paired media-elements have the same `src` (of `HTMLMediaElement`)).
"canplaythrough" event is fired when the ready-state is transited to
HAVE_ENOUGH_DATA. As a result, it will never be fired in this case.
In usual case, if the cache-stream gets suspended from non-suspended, it
will send a "download-suspended-by-cache=true" signal to its paired
media-element when running `MediaCache::Update()`. In fact, all other
cache-streams sharing the same underlying data will send this signal at
the same time if necessary. (Later, once the cache-stream is resumed
from suspended to non-suspended it will send a
"download-suspended-by-cache=false" signal to its paired media-element.
All other cache-streams sharing the same underlying data will do the
same if necessary.) The media-element keeps tracking that signal it
receives. After the first-frame of the media-element is loaded, the
ready-state of the media-element will be transited to HAVE_ENOUGH_DATA
by force if the signal is true. (Otherwise, the ready-state will be
inferred by other information.)
When cloning a cache-stream from another one, the cloned cache-stream is
suspended by default. If it's added to a jammed cache that all of the
cache-streams are suspended since the cache is full, then it never has a
chance to fire the "download-suspended-by-cache" signal. Both its
source-stream and itself have no status-change between suspended and
non-suspended, so `MediaCache::Update` is unable to send the signal.
In this case, we should force the media-element paired with the newly
cloned cache-stream transits its ready-state to HAVE_ENOUGH_DATA, which
follows the existing mechanism, by queueing a status-change-update when
cloning the stream. The status-change-update will be run in the
`MediaCache::Update` and it will check what the signal should be sent.
Once the "download-suspended-by-cache=true" is sent, the
"canplaythrough" event of the media-element can be dispatched after
its first-frame is loaded. (The event listeners can possibly make the
media-element starts playing, which is likely to cause a cache-seek that
can revitalize the cache eventually.)
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1646719#c5
Differential Revision: https://phabricator.services.mozilla.com/D92125
Add some logs that can help us to check how many cache-streams are
suspended and the working status in `MediaCache::Update`
Differential Revision: https://phabricator.services.mozilla.com/D92124
Without this patch, the FrameEncode gtest encoder won't encode the last bit of
data, making the total duration 20ms too short.
When passing EOS we encode the lookahead worth of silence, so this patch also
accounts for that.
Differential Revision: https://phabricator.services.mozilla.com/D91957
Without this patch, there could be an input rate leading to the use of a
resampler *and* framesLeft being 0 (not rounding up). Then we end up with too
little data to feed the resampler, and we fail an assert.
Differential Revision: https://phabricator.services.mozilla.com/D91956
Without this patch, the lookahead silence would be added to the source segment,
which is sampled at the input rate.
The lookahead is in the output rate.
Converting the lookahead to the input rate, and letting our regular encoding
logic convert it back to the output rate can lead to a rounding error for some
input rates. A rounding error here would lead to the last packet being too
short.
Differential Revision: https://phabricator.services.mozilla.com/D91725
There should be no accumulating rounding error here since packet durations are
exactly 200ms which does not lead to a rounding error when converting to
microseconds.
But this is for sanity, since the behavior prior to this patch is exactly how
you get an accumulating rounding error.
Differential Revision: https://phabricator.services.mozilla.com/D91955
Certain logic, like AudioOutputFramesPerPacket(), is hinged off whether a
resampler exists. The resampler is destroyed when encoding is completed, making
such logic flawed from that point. To avoid this potential footgun we can decide
the output rate at construction time, since the input rate is known then.
Differential Revision: https://phabricator.services.mozilla.com/D91953
AudioTrackEncoder uses GetPacketDuration() for signaling upwards that data is
available to be encoded. Data to be encoded is sampled at the input rate while
GetPacketDuration() is the duration in the output rate.
Meanwhile, OpusTrackEncoder uses GetPacketDuration() internally for deciding how
much data to encode. This is after resampling so correctly in the output rate.
To support both these cases, this patch adds NumOutputFramesPerPacket(), modeled
on GetOutputSampleRate(), denoting the packet duration in the output rate.
GetPacketDuration() is renamed to NumInputFramesPerPacket() and changed to be
the packet duration in the input rate.
Differential Revision: https://phabricator.services.mozilla.com/D91952
Without this patch there was a gap between the default-ctor and when real values
got set. If setting a member was forgotten, it would have needed an audit to be
found. With this patch the compiler will make sure all values have been
explicitly handed to the ctor.
mFrameData (nsTArray) becomes Refcountable to allow VP8TrackEncoder to extend
the duration of an EncodedFrame by copying the last frame's ref and constructing
a new EncodedFrame with a longer duration than the last one's.
Differential Revision: https://phabricator.services.mozilla.com/D91724
This was originally handled by EbmlComposer. Since bug 1014393 this was handled
by MediaEncoder. By doing it in OpusTrackEncoder we can avoid reading hardcoded
fields in the opus metadata to get the codec delay value.
Differential Revision: https://phabricator.services.mozilla.com/D91723
When close the event source, it should be responsible to clear up and reset the virtual control interface, rather than doing so by `Media Control Server` via setting some empty results.
Differential Revision: https://phabricator.services.mozilla.com/D92116
The old way to open/close the event source, which is triggered by controller amount change event, is less intuitive, and we do the extra clean up when close the event source by assigning some parameters [1] that causes an issue on Windows where the control interface can't be clear up completely.
Each platform has its own way to clean the interface. For example, on Windows, we can simply call `ISystemMediaTransportControlsDisplayUpdater::ClearAll()`. So calling those functions actually helps nothing. The best way to do that is to ask the event source to do the clean up, rathering than setting those unnecessary parameters.
Therefore, we make it happen closer to when we determine or clear main controller and ask the event source to take a responsible to clean up when it closes.
[1] https://searchfox.org/mozilla-central/rev/35245411b9e8a911fe3f5adb0632c3394f8b4ccb/dom/media/mediacontrol/MediaControlService.cpp#410-413
Differential Revision: https://phabricator.services.mozilla.com/D92115
As we've chosen another way for GeckoView implementation, so `SetControlledTabBrowsingContextId` is no longer needed.
Differential Revision: https://phabricator.services.mozilla.com/D92114
When close the event source, it should be responsible to clear up and reset the virtual control interface, rather than doing so by `Media Control Server` via setting some empty results.
Differential Revision: https://phabricator.services.mozilla.com/D92116
The old way to open/close the event source, which is triggered by controller amount change event, is less intuitive, and we do the extra clean up when close the event source by assigning some parameters [1] that causes an issue on Windows where the control interface can't be clear up completely.
Each platform has its own way to clean the interface. For example, on Windows, we can simply call `ISystemMediaTransportControlsDisplayUpdater::ClearAll()`. So calling those functions actually helps nothing. The best way to do that is to ask the event source to do the clean up, rathering than setting those unnecessary parameters.
Therefore, we make it happen closer to when we determine or clear main controller and ask the event source to take a responsible to clean up when it closes.
[1] https://searchfox.org/mozilla-central/rev/35245411b9e8a911fe3f5adb0632c3394f8b4ccb/dom/media/mediacontrol/MediaControlService.cpp#410-413
Differential Revision: https://phabricator.services.mozilla.com/D92115
As we've chosen another way for GeckoView implementation, so `SetControlledTabBrowsingContextId` is no longer needed.
Differential Revision: https://phabricator.services.mozilla.com/D92114
This avoids us risking an overflow when we convert encrypted media with
subsamples to AnnexB (since that conversion can grow the clear sizes of the
sample). See the test in the preceding patch for an example of how and why this
happens.
Differential Revision: https://phabricator.services.mozilla.com/D92300
Add test code to ensure AnnexB conversions behave as we expect. This adds some
coverage for non-encrypted conversions that we only tested with broader tests
until now. It also adds a test to ensure we don't overflow our subsample sizes
when dealing with encrypted media with very large subsamples. This latter test
covers the issue seen in bug 1618529.
Differential Revision: https://phabricator.services.mozilla.com/D92299
`mBlockOwnersWatermark` is introduced in bug 1366936 for telemetry `MEDIACACHE_BLOCKOWNERS_WATERMARK` but `MEDIACACHE_BLOCKOWNERS_WATERMARK` is removed in bug 1356046
Depends on D92106
Differential Revision: https://phabricator.services.mozilla.com/D92107
`mIndexWatermark` was introduced in bug 1366929 for telemetry `MEDIACACHE_WATERMARK_KB` but `MEDIACACHE_WATERMARK_KB` was removed in bug 1356046
Depends on D92105
Differential Revision: https://phabricator.services.mozilla.com/D92106
These were originally masked due to an exception in
tools/lint/file-whitespace.html for the directory dom/media/tests, but we
don't need an exception for our new location (dom/media/webrtc/tests/mochitests)
if we fix these 3 files.
2 files had bad line endings (Windows vs Unix):
dom/media/webrtc/tests/mochitests/test_getUserMedia_cubebDisabled.html
dom/media/webrtc/tests/mochitests/test_getUserMedia_cubebDisabledFakeStreams.html
1 file had trailing whitespace:
dom/media/webrtc/tests/mochitests/test_peerConnection_threeUnbundledConnections.html
Differential Revision: https://phabricator.services.mozilla.com/D90630
Currently changing the pref `media.hardwaremediakeys.enabled` would only take effect after restarting Firefox, this patch would make it possible to do so during runtime.
Differential Revision: https://phabricator.services.mozilla.com/D91841
This is mostly preparing for the future state where we might have SWGL WR mixed with real hardware webrender, and want a way to lookup the state per-compositor.
Differential Revision: https://phabricator.services.mozilla.com/D92009
This adds a script to vendor libwebrtc and chromium/src/build from upstream
Google or GitHub repositories as well as from local repositories.
Differential Revision: https://phabricator.services.mozilla.com/D91611
This is mostly preparing for the future state where we might have SWGL WR mixed with real hardware webrender, and want a way to lookup the state per-compositor.
Depends on D92008
Differential Revision: https://phabricator.services.mozilla.com/D92009
Fix up includes so AnnexB.h and TestMediaDataEncoder.cpp don't rely on unified
build order. Reformat include lists to match style guide. Rework #include guard
on AnnexB.h to reflect style guide.
Differential Revision: https://phabricator.services.mozilla.com/D91825
An ArrayOfRemoteAudioData pack all its AudioData objects into a single Shmem.
This Shmem will be-reused by the remote decoder over and over.
When used with webaudio, this reduces the number of memory allocation from 100 to 1 for each remote decoder.
Differential Revision: https://phabricator.services.mozilla.com/D91539
The reduce the amount of shmem allocations to 1 instead of 100 when used with webaudio.
We also fix the AudioTrimmer for remote decoders as the trimming information was lost over the IPC serialization.
Another issue corrected is that we no longer crash if the first MediaRawData decoded had an empty size.
Differential Revision: https://phabricator.services.mozilla.com/D91538
Those two objects can be used to pack multiple array of objects into a minimal amount of shmem.
An ArrayOfRemoteByteBuffer will take at most a single shmem and perform a single memory allocation..
Similarly, an ArrayOfMediaRawData will pack multiple MediaRawData in at most 3 shmems (one for each array of bytes a MediaRawData contains).
They are designed to work in combination with a ShmemPool which will own each of the allocated Shmem and so can be re-used over and over.
Differential Revision: https://phabricator.services.mozilla.com/D91537
We don't need `autoplay_would_be_allowed_count` and `autoplay_would_not_be_allowed_count`, so we can remove all related codes.
Differential Revision: https://phabricator.services.mozilla.com/D83227
These were originally masked due to an exception in
tools/lint/file-whitespace.html for the directory dom/media/tests, but we
don't need an exception for our new location (dom/media/webrtc/tests/mochitests)
if we fix these 3 files.
2 files had bad line endings (Windows vs Unix):
dom/media/webrtc/tests/mochitests/test_getUserMedia_cubebDisabled.html
dom/media/webrtc/tests/mochitests/test_getUserMedia_cubebDisabledFakeStreams.html
1 file had trailing whitespace:
dom/media/webrtc/tests/mochitests/test_peerConnection_threeUnbundledConnections.html
Differential Revision: https://phabricator.services.mozilla.com/D90630
These were originally masked due to an exception in
tools/lint/file-whitespace.html for the directory dom/media/tests, but we
don't need an exception for our new location (dom/media/webrtc/tests/mochitests)
if we fix these 3 files.
2 files had bad line endings (Windows vs Unix):
dom/media/webrtc/tests/mochitests/test_getUserMedia_cubebDisabled.html
dom/media/webrtc/tests/mochitests/test_getUserMedia_cubebDisabledFakeStreams.html
1 file had trailing whitespace:
dom/media/webrtc/tests/mochitests/test_peerConnection_threeUnbundledConnections.html
Differential Revision: https://phabricator.services.mozilla.com/D90630
One second is a bit short given that starting bluetooth input devices takes
several seconds on some platforms. Bug 1586370 may have worsened this since we
now process silence while waiting for an input to give us data.
Differential Revision: https://phabricator.services.mozilla.com/D91140
Attempt to lock textures from the same D3D11 devices on multiple threads at once can lead to deadlocks as observed with AMD cards.
Differential Revision: https://phabricator.services.mozilla.com/D91098
Because of D90771, we need media session to notifty its status correctly in order to deactivate the controller. Therefore, when its document becomes inactive (in bfcahce), we should treat media session as inactive and notify it to `MediaStatusManager` in order to clear the active media session if needed.
In addition, add some assertions to ensure we won't modify or set any attributes on media session when its document is inactive.
Differential Revision: https://phabricator.services.mozilla.com/D90926
Currently, we only keep controller active when it has controlled media. That strategy works well for non-media session situation because only controlled media need to listen to media keys.
However, when having media session, thing goes slightly different. When we don't have any controlled media, active media session may still listen to media keys and do the corresponding operation. Therefore, we should keep the active media session being able to receive media keys even if the controlled media has gone and deactivate a controller when it doesn't have controlled media and no active media session.
Example, play a audible media first, then press `next track`, if media session is going to play an inaudible media (which is not a controllable media, so no controlled media is existing), we still want media session to receive and handle media keys for users.
Differential Revision: https://phabricator.services.mozilla.com/D90771
In particular, this removes the code that was limiting the audibility
notifications spam, because this is handled by the AudibilityMonitor.
Differential Revision: https://phabricator.services.mozilla.com/D90433
This is essentially the same code as the interleaved version, but backwards,
since the memory layout is the opposite, we want to take advantage of memory
locality, and only touch audio samples once each.
The `ProcessAudioData` method has been renamed, because of the similarity
between the `AudioData` type and `ProcessAudioData`, since it can now process
`AudioBlock`s.
Differential Revision: https://phabricator.services.mozilla.com/D90431
Because of D90771, we need media session to notifty its status correctly in order to deactivate the controller. Therefore, when its document becomes inactive (in bfcahce), we should treat media session as inactive and notify it to `MediaStatusManager` in order to clear the active media session if needed.
In addition, add some assertions to ensure we won't modify or set any attributes on media session when its document is inactive.
Differential Revision: https://phabricator.services.mozilla.com/D90926
Currently, we only keep controller active when it has controlled media. That strategy works well for non-media session situation because only controlled media need to listen to media keys.
However, when having media session, thing goes slightly different. When we don't have any controlled media, active media session may still listen to media keys and do the corresponding operation. Therefore, we should keep the active media session being able to receive media keys even if the controlled media has gone and deactivate a controller when it doesn't have controlled media and no active media session.
Example, play a audible media first, then press `next track`, if media session is going to play an inaudible media (which is not a controllable media, so no controlled media is existing), we still want media session to receive and handle media keys for users.
Differential Revision: https://phabricator.services.mozilla.com/D90771
Razvan Maries
Paul Adenot
Dan Minor
Andreas Pehrson
Mihai Alexandru Michis
Jean-Yves Avenard
Cosmin Sabou
Narcis Beleuzu
Bryce Seager van Dyk
Dorel Luca
Byron Campen [:bwc]
smolnar
alwu
Matt Woodrow
Bogdan Tara
Aaron Klotz
Kartikaya Gupta
Chun-Min Chang
Jon Bauman
Nico Grunbaum
Kershaw Chang
Noemi Erli
Michael Froman
Csoregi Natalia
Tom Schuster
Butkovits Atila
Sebastian Hengst
Mike Hommey
Jan-Ivar Bruaroey
Nicklas Boman
Simon Giesecke
Daniel Le