For lower resolution videos, we should be able to decode them well without storing too many video frames. So let's keep this mechanism for 4k+ video only.
Differential Revision: https://phabricator.services.mozilla.com/D118402
If the decoded video MDSM received is already late, then it indicates that video decoding is too slow.
If we keep decoding but in the same speed, it's very likely all following decoded videos would still be behind the clock and need to be discarded.
The reader has its own mechanism to seek to the next keyframe, but that will only happen when the time of next keyframe is behind the given time threshold, which is the current time.
When the video has pretty long keyframe interval (eg. 5s), then that mechanism won't be triggered because the even if the decoded video frames are late, the next keyframe still far exceeds the current time, so the reader doesn't think it's necessary to skip.
Therefore, MDSM should force the reader to seek to the next keyframe directly, because it know both the decoded time of video frames and the current playback time.
Differential Revision: https://phabricator.services.mozilla.com/D117242
The purpose of this patch is to create a consistent clock timeline among different media sinks.
Having a consistent clock timeline can help the newly created media sink to understand audio data's "adjusted" time, which can prevent the sink from inserting silence if the sink incorrectly think there is a gap between data and sink start time.
However, once seeking completed, we should use the media time that indicates to the new position, instead of using the previous clock time.
Eg.
Audio duration = 10, and has seamlessly looped 5 times. Then the start time of the sample in 2s in the 6th looping would be 2 + 5*10 = 52.
If we switch to the new sink, and use media time (which is 2s) as a start time to start a new sink, then the new sink would think there is a gap between 2s and 52s and push a lot of silence into the data. [1]
[1] https://searchfox.org/mozilla-central/rev/9b430bb1a11d7152cab2af4574f451ffb906b052/dom/media/mediasink/AudioSink.cpp#390-426
Differential Revision: https://phabricator.services.mozilla.com/D113548
When enabling our MediaTrack implementation (which we don't plan to by default,
NB) and disabling all audio tracks and unselecting all video tracks while having
an active captureStream leads to having no output tracks in DecodedStream.
In this case, DecodedStream doesn't know which graph to use for creating the
intermediary tracks it feeds data to. We don't want to resort to the default
graph either, since two graphs on different clocks could then race each other.
With this patch we plumb down a SharedDummyTrack from the media element where
the captureStream was triggered, through MediaDecoder, to DecodedStream. The
SharedDummyTrack guarantees to keep the graph alive, and holds the graph used
for the output tracks.
Differential Revision: https://phabricator.services.mozilla.com/D99822
Implement the `MediaDecoder::Suspend/Resume` to shutdown and recreate the MediaSink. This is important the page is refreshed because the existing tree does not shutdown immediatelly. It stays alive but remains suspended thus it is necessary to clean up the low-level system resources. In order to support the suspended state with the existing work flow, if MediaSink is suspended it is disallowed to start it or change the sink id. Recreating the MediaSink is possible but starting it is restricted until resume.
Differential Revision: https://phabricator.services.mozilla.com/D63831
--HG--
extra : moz-landing-system : lando
This converts `nsAutoPtr` usage in dom/media to `UniquePtr`. Beyond just a
search and replace we also needed to update assignment and access of the
`UniquePtr`s. This falls into a few categories:
- Assignment from a newly constructed object switches to `MakeUnique`
- Assignment from a raw ptr switches to `UniquePtr::reset`
- Handing out a raw ptr now requires `UniquePtr::get`
- Uses `UniquePtr::release` rather than `nsAutoPtr::forget`
- A few spots are updated to return a `UniquePtr` rather than a raw ptr
Differential Revision: https://phabricator.services.mozilla.com/D60084
--HG--
extra : moz-landing-system : lando
There are multiple seeking states in `MDSM`, it would be good to see the full name of the seeking state on the debug log to know what seeking state we're using.
Differential Revision: https://phabricator.services.mozilla.com/D56823
--HG--
extra : moz-landing-system : lando
As we added new files, which affects the build bundle created by unified build. Therefore, add missing scope definition and included header to pass the build.
Differential Revision: https://phabricator.services.mozilla.com/D52430
--HG--
extra : moz-landing-system : lando
As we added new files, which affects the build bundle created by unified build. Therefore, add missing scope definition and included header to pass the build.
Differential Revision: https://phabricator.services.mozilla.com/D52430
--HG--
extra : moz-landing-system : lando
This patch does the following:
- Makes cloneElementVisually() return a promise
- Plumbs an event from the MediaDecoderStateMachine's VideoSink to
HTMLVideoElement
- Hooks the event up to resolve the promise from cloneElementVisually()
- Updates tests and their expectations.
Differential Revision: https://phabricator.services.mozilla.com/D53831
--HG--
extra : moz-landing-system : lando
This better follows how MediaDecoder and MediaDecoderStateMachine was
architected.
Differential Revision: https://phabricator.services.mozilla.com/D53710
--HG--
extra : moz-landing-system : lando
This patch does the following:
- Makes cloneElementVisually() return a promise
- Plumbs an event from the MediaDecoderStateMachine's VideoSink to
HTMLVideoElement
- Hooks the event up to resolve the promise from cloneElementVisually()
- Updates tests and their expectations.
Differential Revision: https://phabricator.services.mozilla.com/D53831
--HG--
extra : moz-landing-system : lando
This better follows how MediaDecoder and MediaDecoderStateMachine was
architected.
Differential Revision: https://phabricator.services.mozilla.com/D53710
--HG--
extra : moz-landing-system : lando
This patches does several minor things:
- Moves SetSink (from setSinkid) to automatic coalescing of multiple calls
through a Canonical/Mirror setup instead of a manual atomic counter.
- Simplifies the logic for when to update the sink in SetSink.
- Removes PlaybackParams as a general MediaSink property, as it only contains
audio params.
- Makes PlaybackParams an internal AudioSink concept, that AudioSinkWrapper
knows about.
- Ensures mMediaSink is only accessed on the decoder TaskQueue, to allow
accessing mirrored members when creating it.
Differential Revision: https://phabricator.services.mozilla.com/D52043
--HG--
extra : moz-landing-system : lando
This patch removes the responsibility of js-facing MediaStreamTracks from the
MediaDecoder stack, and moves the machinery for setting up DecodedStream to
higher order functions like state mirroring and watchables.
OutputStreamManager is completely gone, since it was designed to manage
MediaStreamTracks across multiple output streams for a single decoder,
on main thread. HTMLMediaElement took over its task in the previous patch.
The MediaDecoderStateMachine now has three control points for capturing:
- mOutputCaptured, which, if true, will capture all decoded data into
mOutputTracks. If this is set, but mOutputTracks is empty, we are still
waiting for tracks, and DecodedStream will not play any data. When tracks are
set, a new DecodedStream is created that will play data through
SourceMediaTracks piped into mOutputTracks.
- mOutputTracks, which is the set of tracks data is captured into, for
forwarding to all the output tracks the media element is managing. This set of
tracks is managed by the MediaDecoder owner, and must contain one audio track
if the decoder is decoding audio, and one video track if the decoder is
decoding video. It may be empty since output can be captured before metadata
is loaded, or playback has ended.
- mOutputPrincipal, which is the principal of the decoded data. All data sent
into SourceMediaTracks is tagged with this principal.
Differential Revision: https://phabricator.services.mozilla.com/D52042
--HG--
extra : moz-landing-system : lando
This patches does several minor things:
- Moves SetSink (from setSinkid) to automatic coalescing of multiple calls
through a Canonical/Mirror setup instead of a manual atomic counter.
- Simplifies the logic for when to update the sink in SetSink.
- Removes PlaybackParams as a general MediaSink property, as it only contains
audio params.
- Makes PlaybackParams an internal AudioSink concept, that AudioSinkWrapper
knows about.
- Ensures mMediaSink is only accessed on the decoder TaskQueue, to allow
accessing mirrored members when creating it.
Differential Revision: https://phabricator.services.mozilla.com/D52043
--HG--
extra : moz-landing-system : lando
This patch removes the responsibility of js-facing MediaStreamTracks from the
MediaDecoder stack, and moves the machinery for setting up DecodedStream to
higher order functions like state mirroring and watchables.
OutputStreamManager is completely gone, since it was designed to manage
MediaStreamTracks across multiple output streams for a single decoder,
on main thread. HTMLMediaElement took over its task in the previous patch.
The MediaDecoderStateMachine now has three control points for capturing:
- mOutputCaptured, which, if true, will capture all decoded data into
mOutputTracks. If this is set, but mOutputTracks is empty, we are still
waiting for tracks, and DecodedStream will not play any data. When tracks are
set, a new DecodedStream is created that will play data through
SourceMediaTracks piped into mOutputTracks.
- mOutputTracks, which is the set of tracks data is captured into, for
forwarding to all the output tracks the media element is managing. This set of
tracks is managed by the MediaDecoder owner, and must contain one audio track
if the decoder is decoding audio, and one video track if the decoder is
decoding video. It may be empty since output can be captured before metadata
is loaded, or playback has ended.
- mOutputPrincipal, which is the principal of the decoded data. All data sent
into SourceMediaTracks is tagged with this principal.
Differential Revision: https://phabricator.services.mozilla.com/D52042
--HG--
extra : moz-landing-system : lando
This is inherently large, because modifying these bits of DOMMediaStream and
MediaStreamTrack affects all consumers and producers of all DOMMediaStreams and
MediaStreamTracks.
Things are generally much simpler now.
Producers of tracks now create a MediaStream in the graph, add it to a
MediaStreamTrackSource subclass that takes ownership of it, and add the source
to a MediaStreamTrack. Should the producer need a DOMMediaStream it is now much
simpler to create as the only thing needed is the current window. The stream is
a rather simple wrapper around an array of MediaStreamTracks.
HTMLMediaElement is still not as straight forward as other consumers since it
consumes the DOMMediaStream directly, as opposed to a set of tracks.
The new MediaStreamRenderer helper class helps bridge the gap between this fact
and the new track-based MediaStreamGraph interface, as it needs to juggle
registering multiple audio tracks for audio output. This hooks into existing
HTMLMediaElement logic and brings a welcome simplification to all the glue
previously needed there.
Differential Revision: https://phabricator.services.mozilla.com/D37934
--HG--
extra : moz-landing-system : lando
This patch structurizes the media debug information via webidl dictionaries
that are returned by HTMLMediaElement::GetMozRequestDebugInfo() and
MediaSource::GetMozDebugReaderData().
Differential Revision: https://phabricator.services.mozilla.com/D27893
--HG--
extra : moz-landing-system : lando
The normal looping process is that, goes to `completed` state first, notify playback ended, and finally media element would call seek to the start position in order to start looping again.
However, if we're in the seamless looping mode, we can stay in `loopingDecoding` state and repeating the looping without going to other states. Otherwise, we should go to `completed` state if decoding has ended.
Differential Revision: https://phabricator.services.mozilla.com/D30307
--HG--
extra : moz-landing-system : lando
This patch structurizes the media debug information via webidl dictionaries
that are returned by HTMLMediaElement::GetMozRequestDebugInfo() and
MediaSource::GetMozDebugReaderData().
Differential Revision: https://phabricator.services.mozilla.com/D27893
--HG--
extra : moz-landing-system : lando
This moves the responsibility for creating MediaStreamTracks from
DecodedStream::Start to MediaDecoder. This let's MediaDecoder create them as
soon as metadata is known. This gives the application guarantees on when tracks
can be expected to exist, and aligns with the spec that says they should be
created when metadata is known.
Differential Revision: https://phabricator.services.mozilla.com/D28473
--HG--
extra : moz-landing-system : lando
The VideoSink shares the AudioSink's own EndedPromise to notify its user that it has ended. As such, the MozPromise used must be non-exclusive.
Using the GenericPromise for such purpose only hid that requirement.
We also remove the MediaSink from the media namespace, and clarify the naming of some arguments and class members to accurately describe what they do.
Differential Revision: https://phabricator.services.mozilla.com/D14024
--HG--
extra : moz-landing-system : lando
This removes DecodedStream's use of MediaStreamListener in favor of
MediaStreamTrackListener. This change has however rippled through to a lot
more cleanup, per below.
This moves the MediaStreamTrack lifetime ownership for captured
HTMLMediaElements from the media element to DecodedStream, where the
MediaStreamGraph-side tracks are already created and ended today.
This makes MediaStreamTrack creation explicit across the entire codebase and
lets us remove the MediaStreamTrackSourceGetter class and the infrastructure
of adding MediaStreamTracks after they've already been created in the graph
from DOMMediaStream.
With track ownership, and thus TrackID allocation ownership, happening
exclusively in DecodedStream for its output tracks, we also stop throwing
away and recreating the SourceMediaStream to which we feed data on seek.
This is one step closer to fixing bug 1172394 and spec compliance of
HTMLMediaElement.captureStream().
Differential Revision: https://phabricator.services.mozilla.com/D12273
--HG--
extra : moz-landing-system : lando
This state is used to handle decoding when the media is in seamless looping. The advantage to create a new state is that we can totally separate looping and non-looping codes and then they won't affect each other anymore.
The new state will be responsible for
(1) time adjustment
(2) handle EOS and seek to the first sample
Differential Revision: https://phabricator.services.mozilla.com/D9186
--HG--
extra : moz-landing-system : lando
Same approach as the other bug, mostly replacing automatically by removing
'using mozilla::Forward;' and then:
s/mozilla::Forward/std::forward/
s/Forward</std::forward</
The only file that required manual fixup was TestTreeTraversal.cpp, which had
a class called TestNodeForward with template parameters :)
MozReview-Commit-ID: A88qFG5AccP
alwu
Csoregi Natalia
Brendan Early
Andreas Pehrson
Simon Giesecke
Alex Chronopoulos
Eric Rahm
Noemi Erli
Daniel Varga
Gurzau Raul
Tarek Ziadé
shindli
Mike Conley
Ehsan Akhgari
Jean-Yves Avenard
Sylvestre Ledru
Dorel Luca
Bogdan Tara
Andreas Pehrson
Emilio Cobos Álvarez