This patch removes the responsibility of js-facing MediaStreamTracks from the
MediaDecoder stack, and moves the machinery for setting up DecodedStream to
higher order functions like state mirroring and watchables.
OutputStreamManager is completely gone, since it was designed to manage
MediaStreamTracks across multiple output streams for a single decoder,
on main thread. HTMLMediaElement took over its task in the previous patch.
The MediaDecoderStateMachine now has three control points for capturing:
- mOutputCaptured, which, if true, will capture all decoded data into
mOutputTracks. If this is set, but mOutputTracks is empty, we are still
waiting for tracks, and DecodedStream will not play any data. When tracks are
set, a new DecodedStream is created that will play data through
SourceMediaTracks piped into mOutputTracks.
- mOutputTracks, which is the set of tracks data is captured into, for
forwarding to all the output tracks the media element is managing. This set of
tracks is managed by the MediaDecoder owner, and must contain one audio track
if the decoder is decoding audio, and one video track if the decoder is
decoding video. It may be empty since output can be captured before metadata
is loaded, or playback has ended.
- mOutputPrincipal, which is the principal of the decoded data. All data sent
into SourceMediaTracks is tagged with this principal.
Differential Revision: https://phabricator.services.mozilla.com/D52042
--HG--
extra : moz-landing-system : lando
This reworks how media element captureStream works by removing the differences
between MediaStream and MediaDecoder capture. MediaDecoder capture will be
refactored so that ownership of MediaStreamTracks lies with the media element
instead of the OutputStreamManager. The internal MediaDecoder parts happen in a
later patch.
The new API for capturing a MediaDecoder involves a boolean on/off toggle, the
output tracks the decoder pipes data to, and the principal that data is tagged
with. If capturing is on but there are no output tracks, playback will not
happen, to ensure that no data gets accidentally skipped in the output tracks
while captured.
This also changes the logic for setting up MediaElementTrackSources in
HTMLMediaElement so it's triggered by the WatchManager and thus run in tail
dispatched runnables.
Differential Revision: https://phabricator.services.mozilla.com/D52040
--HG--
extra : moz-landing-system : lando
HTMLMediaElement avoid creating new tracks in MetadataLoaded when it has already
created some, so there should be no side effect to this patch.
Differential Revision: https://phabricator.services.mozilla.com/D52037
--HG--
extra : moz-landing-system : lando
This patches does several minor things:
- Moves SetSink (from setSinkid) to automatic coalescing of multiple calls
through a Canonical/Mirror setup instead of a manual atomic counter.
- Simplifies the logic for when to update the sink in SetSink.
- Removes PlaybackParams as a general MediaSink property, as it only contains
audio params.
- Makes PlaybackParams an internal AudioSink concept, that AudioSinkWrapper
knows about.
- Ensures mMediaSink is only accessed on the decoder TaskQueue, to allow
accessing mirrored members when creating it.
Differential Revision: https://phabricator.services.mozilla.com/D52043
--HG--
extra : moz-landing-system : lando
This patch removes the responsibility of js-facing MediaStreamTracks from the
MediaDecoder stack, and moves the machinery for setting up DecodedStream to
higher order functions like state mirroring and watchables.
OutputStreamManager is completely gone, since it was designed to manage
MediaStreamTracks across multiple output streams for a single decoder,
on main thread. HTMLMediaElement took over its task in the previous patch.
The MediaDecoderStateMachine now has three control points for capturing:
- mOutputCaptured, which, if true, will capture all decoded data into
mOutputTracks. If this is set, but mOutputTracks is empty, we are still
waiting for tracks, and DecodedStream will not play any data. When tracks are
set, a new DecodedStream is created that will play data through
SourceMediaTracks piped into mOutputTracks.
- mOutputTracks, which is the set of tracks data is captured into, for
forwarding to all the output tracks the media element is managing. This set of
tracks is managed by the MediaDecoder owner, and must contain one audio track
if the decoder is decoding audio, and one video track if the decoder is
decoding video. It may be empty since output can be captured before metadata
is loaded, or playback has ended.
- mOutputPrincipal, which is the principal of the decoded data. All data sent
into SourceMediaTracks is tagged with this principal.
Differential Revision: https://phabricator.services.mozilla.com/D52042
--HG--
extra : moz-landing-system : lando
This reworks how media element captureStream works by removing the differences
between MediaStream and MediaDecoder capture. MediaDecoder capture will be
refactored so that ownership of MediaStreamTracks lies with the media element
instead of the OutputStreamManager. The internal MediaDecoder parts happen in a
later patch.
The new API for capturing a MediaDecoder involves a boolean on/off toggle, the
output tracks the decoder pipes data to, and the principal that data is tagged
with. If capturing is on but there are no output tracks, playback will not
happen, to ensure that no data gets accidentally skipped in the output tracks
while captured.
This also changes the logic for setting up MediaElementTrackSources in
HTMLMediaElement so it's triggered by the WatchManager and thus run in tail
dispatched runnables.
Differential Revision: https://phabricator.services.mozilla.com/D52040
--HG--
extra : moz-landing-system : lando
HTMLMediaElement avoid creating new tracks in MetadataLoaded when it has already
created some, so there should be no side effect to this patch.
Differential Revision: https://phabricator.services.mozilla.com/D52037
--HG--
extra : moz-landing-system : lando
Only SeekToNextFrame cares about promises, but prior to this patch the common
method HTMLMediaElement::Seek() would always return a promise.
When the caller was not SeekToNextFrame (e.g., SetCurrentTime, FastSeek), the
promise would end up *not* being exposed. When later rejected, we would catch
this and write an error to the js console.
This patch lifts the handling of the promise out of Seek() and into
SeekToNextFrame() so that we avoid creating promises that would not get exposed
to js.
Differential Revision: https://phabricator.services.mozilla.com/D42511
--HG--
extra : moz-landing-system : lando
Only SeekToNextFrame cares about promises, but prior to this patch the common
method HTMLMediaElement::Seek() would always return a promise.
When the caller was not SeekToNextFrame (e.g., SetCurrentTime, FastSeek), the
promise would end up *not* being exposed. When later rejected, we would catch
this and write an error to the js console.
This patch lifts the handling of the promise out of Seek() and into
SeekToNextFrame() so that we avoid creating promises that would not get exposed
to js.
Differential Revision: https://phabricator.services.mozilla.com/D42511
--HG--
extra : moz-landing-system : lando
Create an event in MediaFormatReader the will signal to the HTMLMediaElement to initiate a new storing according to the latest VideoInfo. Also when the application is shutting down, trigger a new storing early enough, before all the events are disconnected.
Differential Revision: https://phabricator.services.mozilla.com/D38316
--HG--
extra : moz-landing-system : lando
This is inherently large, because modifying these bits of DOMMediaStream and
MediaStreamTrack affects all consumers and producers of all DOMMediaStreams and
MediaStreamTracks.
Things are generally much simpler now.
Producers of tracks now create a MediaStream in the graph, add it to a
MediaStreamTrackSource subclass that takes ownership of it, and add the source
to a MediaStreamTrack. Should the producer need a DOMMediaStream it is now much
simpler to create as the only thing needed is the current window. The stream is
a rather simple wrapper around an array of MediaStreamTracks.
HTMLMediaElement is still not as straight forward as other consumers since it
consumes the DOMMediaStream directly, as opposed to a set of tracks.
The new MediaStreamRenderer helper class helps bridge the gap between this fact
and the new track-based MediaStreamGraph interface, as it needs to juggle
registering multiple audio tracks for audio output. This hooks into existing
HTMLMediaElement logic and brings a welcome simplification to all the glue
previously needed there.
Differential Revision: https://phabricator.services.mozilla.com/D37934
--HG--
extra : moz-landing-system : lando
This requires replacing inclusions of it with inclusions of more specific prefs
files.
The exception is that StaticPrefsAll.h, which is equivalent to StaticPrefs.h,
and is used in `Codegen.py` because doing something smarter is tricky and
suitable for a follow-up. As a result, any change to StaticPrefList.yaml will
still trigger recompilation of all the generated DOM bindings files, but that's
still a big improvement over trigger recompilation of every file that uses
static prefs.
Most of the changes in this commit are very boring. The only changes that are
not boring are modules/libpref/*, Codegen.py, and ServoBindings.toml.
Differential Revision: https://phabricator.services.mozilla.com/D39138
--HG--
extra : moz-landing-system : lando
This also removes the following prefs, because they're unused:
- media.autoplay.allow-muted pref
- media.autoplay.blackList-override-default
Differential Revision: https://phabricator.services.mozilla.com/D36396
--HG--
extra : rebase_source : 0570540496302b3efedadf4d5115ee5422d5c279
This patch structurizes the media debug information via webidl dictionaries
that are returned by HTMLMediaElement::GetMozRequestDebugInfo() and
MediaSource::GetMozDebugReaderData().
Differential Revision: https://phabricator.services.mozilla.com/D27893
--HG--
extra : moz-landing-system : lando
This patch structurizes the media debug information via webidl dictionaries
that are returned by HTMLMediaElement::GetMozRequestDebugInfo() and
MediaSource::GetMozDebugReaderData().
Differential Revision: https://phabricator.services.mozilla.com/D27893
--HG--
extra : moz-landing-system : lando
Per the discussion in:
https://groups.google.com/d/msg/mozilla.dev.platform/P79pwa9z5m8/iPYPAWPHCAAJ
They should be CamelCase, and that's what most of them already do. This converts
the rest, which are a few.
For the ones that already used `e` or `k` prefixes, I've mostly done:
for file in $(rg Type::e layout | cut -d : -f 1 | sort | uniq); do sed -i 's#Type::e#Type::#g' $file; done
For the ones that used uppercase, I've removed the prefix if it was already in
the type name, and turn them into CamelCase.
Depends on D28680
Differential Revision: https://phabricator.services.mozilla.com/D28681
--HG--
extra : moz-landing-system : lando
This moves the responsibility for creating MediaStreamTracks from
DecodedStream::Start to MediaDecoder. This let's MediaDecoder create them as
soon as metadata is known. This gives the application guarantees on when tracks
can be expected to exist, and aligns with the spec that says they should be
created when metadata is known.
Differential Revision: https://phabricator.services.mozilla.com/D28473
--HG--
extra : moz-landing-system : lando
Summary: Really sorry for the size of the patch. It's mostly automatic
s/nsIDocument/Document/ but I had to fix up in a bunch of places manually to
add the right namespacing and such.
Overall it's not a very interesting patch I think.
nsDocument.cpp turns into Document.cpp, nsIDocument.h into Document.h and
nsIDocumentInlines.h into DocumentInlines.h.
I also changed a bunch of nsCOMPtr usage to RefPtr, but not all of it.
While fixing up some of the bits I also removed some unneeded OwnerDoc() null
checks and such, but I didn't do anything riskier than that.
If video has not been within the potential visible range (which is larger than viewport) yet, its visibility state won't
be updated and would stay in 'UNTRACK'. As those kinds of video are still invisible to users, we don't need to decode
any video frames, we can suspend their video decoding until they're going to be visible.
Differential Revision: https://phabricator.services.mozilla.com/D13804
--HG--
extra : moz-landing-system : lando
If video has not been within the potential visible range (which is larger than viewport) yet, its visibility state won't
be updated and would stay in 'UNTRACK'. As those kinds of video are still invisible to users, we don't need to decode
any video frames, we can suspend their video decoding until they're going to be visible.
Differential Revision: https://phabricator.services.mozilla.com/D13804
--HG--
extra : moz-landing-system : lando
This removes DecodedStream's use of MediaStreamListener in favor of
MediaStreamTrackListener. This change has however rippled through to a lot
more cleanup, per below.
This moves the MediaStreamTrack lifetime ownership for captured
HTMLMediaElements from the media element to DecodedStream, where the
MediaStreamGraph-side tracks are already created and ended today.
This makes MediaStreamTrack creation explicit across the entire codebase and
lets us remove the MediaStreamTrackSourceGetter class and the infrastructure
of adding MediaStreamTracks after they've already been created in the graph
from DOMMediaStream.
With track ownership, and thus TrackID allocation ownership, happening
exclusively in DecodedStream for its output tracks, we also stop throwing
away and recreating the SourceMediaStream to which we feed data on seek.
This is one step closer to fixing bug 1172394 and spec compliance of
HTMLMediaElement.captureStream().
Differential Revision: https://phabricator.services.mozilla.com/D12273
--HG--
extra : moz-landing-system : lando
Nobody is mirroring to 'mLogicallySeeking', so we can just use Watchable instead.
Differential Revision: https://phabricator.services.mozilla.com/D9642
--HG--
extra : moz-landing-system : lando
Add some memory usage information to the Performance counters and make everything asynchronous.
Differential Revision: https://phabricator.services.mozilla.com/D7984
--HG--
extra : moz-landing-system : lando
Andreas Pehrson
Csoregi Natalia
Ryan VanderMeulen
Gurzau Raul
Karl Tomlinson
Coroiu Cristina
Sylvestre Ledru
Alex Chronopoulos
Nicholas Nethercote
Tarek Ziadé
shindli
Emilio Cobos Álvarez
Mike Conley
Jean-Yves Avenard
Ehsan Akhgari
alwu
John Lin
Cosmin Sabou
Nathan Froyd