As we would calculate time difference between frames by checking their timestamp, so we would always ensure that image has valid timestamp.
Differential Revision: https://phabricator.services.mozilla.com/D48034
--HG--
extra : moz-landing-system : lando
This change mainly removes the `mTracks` member from MediaStream and moves all
associated members up a level, so that a MediaStream in practice represents a
single track.
Classes will be renamed in a future patch to reflect this.
Other changes include:
The new `mEnded` member of MediaStream changes meaning to only become true when
all data in the stream has been processed. It stems from
StreamTracks::Track::mEnded which used to become true as soon as the last bit of
data had been added to a track, and there could still be data in the track that
would get processed in future iterations. We are moving towards not having any
future data in tracks, which is why this change is ok to make -- keeping the old
behavior will soon not make sense.
TrackUnionStream is changed to no longer take a list of streams as input and
forward the union of their tracks to itself. Instead it's limited to having one
track as input at a time.
The autofinishing functionality that TrackUnionStream had before has been
transformed into an autoending functionality to allow it to defer ending until
its been told that it's ok to end through the control API. This lets a single
TrackUnionStream span the lifetime of multiple inputs, which will be useful for
making DecodedStream spec compliant with HTMLMediaElement::CaptureStream(), and
for implementing the currently discussed MediaRecorder::ReplaceTrack(), to name
a few potential use cases.
AudioNodeStreams used to only have a track (and thus an AudioSegment) if the
EXTERNAL_OUTPUT flag was enabled on them. With all MediaStreams now representing
a track, AudioNodeStreams inherently have an AudioSegment as a member. It is
however only used with data if the EXTERNAL_OUTPUT flag is enabled.
Differential Revision: https://phabricator.services.mozilla.com/D45821
--HG--
extra : moz-landing-system : lando
This change mainly removes the `mTracks` member from MediaStream and moves all
associated members up a level, so that a MediaStream in practice represents a
single track.
Classes will be renamed in a future patch to reflect this.
Other changes include:
The new `mEnded` member of MediaStream changes meaning to only become true when
all data in the stream has been processed. It stems from
StreamTracks::Track::mEnded which used to become true as soon as the last bit of
data had been added to a track, and there could still be data in the track that
would get processed in future iterations. We are moving towards not having any
future data in tracks, which is why this change is ok to make -- keeping the old
behavior will soon not make sense.
TrackUnionStream is changed to no longer take a list of streams as input and
forward the union of their tracks to itself. Instead it's limited to having one
track as input at a time.
The autofinishing functionality that TrackUnionStream had before has been
transformed into an autoending functionality to allow it to defer ending until
its been told that it's ok to end through the control API. This lets a single
TrackUnionStream span the lifetime of multiple inputs, which will be useful for
making DecodedStream spec compliant with HTMLMediaElement::CaptureStream(), and
for implementing the currently discussed MediaRecorder::ReplaceTrack(), to name
a few potential use cases.
AudioNodeStreams used to only have a track (and thus an AudioSegment) if the
EXTERNAL_OUTPUT flag was enabled on them. With all MediaStreams now representing
a track, AudioNodeStreams inherently have an AudioSegment as a member. It is
however only used with data if the EXTERNAL_OUTPUT flag is enabled.
Differential Revision: https://phabricator.services.mozilla.com/D45821
--HG--
extra : moz-landing-system : lando
Payloads will serialize themselves into a `BlocksRingBuffer` entry when first
captured.
Later they will be deserialized, to stream JSON for the output profile.
Differential Revision: https://phabricator.services.mozilla.com/D43428
--HG--
extra : moz-landing-system : lando
Payloads will serialize themselves into a `BlocksRingBuffer` entry when first
captured.
Later they will be deserialized, to stream JSON for the output profile.
Differential Revision: https://phabricator.services.mozilla.com/D43428
--HG--
extra : moz-landing-system : lando
All calls to `profiler_add_marker()` (outside of the profilers code) are
now replaced by either:
- `PROFILER_ADD_MARKER(name, categoryPair)`
- `PROFILER_ADD_MARKER_WITH_PAYLOAD(name, categoryPair, TypeOfMarkerPayload,
(payload, ..., arguments))`
This makes all calls consistent, and they won't need to prefix the category pair
with `JS::ProfilingCategoryPair::`.
Also it will make it easier to add (and later remove) internal-profiling
instrumentation (bug 1576550), and to replace heap-allocated payloads with
stack-allocated ones (bug 1576555).
Differential Revision: https://phabricator.services.mozilla.com/D43588
--HG--
extra : moz-landing-system : lando
To accomodate this, MediaStreamTrackListener::NotifyEnded/Removed get an
additional aGraph argument. NotifyEnabledStateChanged is getting it too, for
completion.
Differential Revision: https://phabricator.services.mozilla.com/D43667
--HG--
extra : moz-landing-system : lando
It would cause an assertion failure when OutputStreamManager was released on
main thread. It could be wrapped in an nsMainThreadPtrHandle instead, but that's
exactly what mPrincipalHandle is, so we can use that for both needs.
Differential Revision: https://phabricator.services.mozilla.com/D40789
--HG--
extra : moz-landing-system : lando
This hasn't been needed since the last larger refactor of DecodedStream
(bug 1423241), but got incorporated wrongly with bug 1493613.
When DecodedStream is Stop()ed and then Start()ed, a track is added to the graph
with a playout position starting at 0, and mStartTime is set to the starting
position, i.e., the seeked position in case of a seek or 0 if decoding from the
start of a file.
Hence, the reported position by DecodedStream should be mStartTime plus the last
reported output time of the tracks.
mStreamTimeOffset was offseting the tracks' reported output time further, so
DecodedStream was reporting a too large number.
Differential Revision: https://phabricator.services.mozilla.com/D40626
--HG--
extra : moz-landing-system : lando
This is inherently large, because modifying these bits of DOMMediaStream and
MediaStreamTrack affects all consumers and producers of all DOMMediaStreams and
MediaStreamTracks.
Things are generally much simpler now.
Producers of tracks now create a MediaStream in the graph, add it to a
MediaStreamTrackSource subclass that takes ownership of it, and add the source
to a MediaStreamTrack. Should the producer need a DOMMediaStream it is now much
simpler to create as the only thing needed is the current window. The stream is
a rather simple wrapper around an array of MediaStreamTracks.
HTMLMediaElement is still not as straight forward as other consumers since it
consumes the DOMMediaStream directly, as opposed to a set of tracks.
The new MediaStreamRenderer helper class helps bridge the gap between this fact
and the new track-based MediaStreamGraph interface, as it needs to juggle
registering multiple audio tracks for audio output. This hooks into existing
HTMLMediaElement logic and brings a welcome simplification to all the glue
previously needed there.
Differential Revision: https://phabricator.services.mozilla.com/D37934
--HG--
extra : moz-landing-system : lando
When unknown, we rely on the picture height and assume that anything less than 720p is 601 and 709 otherwise. It's not perfect but it's the best we can do.
Differential Revision: https://phabricator.services.mozilla.com/D39275
--HG--
extra : moz-landing-system : lando
When unknown, we rely on the picture height and assume that anything less than 720p is 601 and 709 otherwise. It's not perfect but it's the best we can do.
Differential Revision: https://phabricator.services.mozilla.com/D39275
--HG--
extra : moz-landing-system : lando
When unknown, we rely on the picture height and assume that anything less than 720p is 601 and 709 otherwise. It's not perfect but it's the best we can do.
Differential Revision: https://phabricator.services.mozilla.com/D39275
--HG--
extra : moz-landing-system : lando
This requires replacing inclusions of it with inclusions of more specific prefs
files.
The exception is that StaticPrefsAll.h, which is equivalent to StaticPrefs.h,
and is used in `Codegen.py` because doing something smarter is tricky and
suitable for a follow-up. As a result, any change to StaticPrefList.yaml will
still trigger recompilation of all the generated DOM bindings files, but that's
still a big improvement over trigger recompilation of every file that uses
static prefs.
Most of the changes in this commit are very boring. The only changes that are
not boring are modules/libpref/*, Codegen.py, and ServoBindings.toml.
Differential Revision: https://phabricator.services.mozilla.com/D39138
--HG--
extra : moz-landing-system : lando
These files exist because they were the proof-of-concept first step for
splitting the static prefs header files. Now that those header files can be
generated from a script, we need to move the `accessibility.*` prefs into the
YAML file.
Differential Revision: https://phabricator.services.mozilla.com/D39132
--HG--
extra : moz-landing-system : lando
This implements the machinery for the splitting of static prefs headers, and
uses it for a single header. #includes are used in such a way that the amount
of boilerplate for each static prefs header file is minimal.
Future patches will split the remaining prefs into more header files.
Differential Revision: https://phabricator.services.mozilla.com/D36154
--HG--
rename : modules/libpref/StaticPrefs.h => modules/libpref/StaticPrefsBase.h
rename : modules/libpref/StaticPrefs.h => modules/libpref/init/StaticPrefListBegin.h
extra : moz-landing-system : lando
MediaInputPort::Destroy must always be called, or it will not remove its strong
ref to the MediaStreamGraph. The guard is not even needed, since
MediaInputPort::Disconnect is idempotent.
Differential Revision: https://phabricator.services.mozilla.com/D36571
--HG--
extra : moz-landing-system : lando
This also removes the following prefs, because they're unused:
- media.autoplay.allow-muted pref
- media.autoplay.blackList-override-default
Differential Revision: https://phabricator.services.mozilla.com/D36396
--HG--
extra : rebase_source : 0570540496302b3efedadf4d5115ee5422d5c279
This patch structurizes the media debug information via webidl dictionaries
that are returned by HTMLMediaElement::GetMozRequestDebugInfo() and
MediaSource::GetMozDebugReaderData().
Differential Revision: https://phabricator.services.mozilla.com/D27893
--HG--
extra : moz-landing-system : lando
This assert can be racy if the following sequence happens:
SetPlaying() -- watch manager schedules SendData()
Stop() -- unwatches mPlaying, resets mStartTime
SendData() -- fails the assert
SendData handles the case where mStartTime is unset at runtime already through the mData check, further making this assert unnecessary.
Depends on D32112
Differential Revision: https://phabricator.services.mozilla.com/D32113
--HG--
extra : moz-landing-system : lando
The following sequence in DecodedStream could lead to a stall for a captured
media element:
- MediaDecoder decodes some data, DecodedStream sends it out
- MediaDecoder pauses for buffering, DecodedStream sees SetPlaying(false)
- MediaDecoder decodes until the end, DecodedStream doesn't send since it's
not playing
- MediaDecoder starts playing again, DecodedStream sees SetPlaying(true)
At the last step above, SetPlaying(true) doesn't trigger SendData() so no data
is sent out. Since all data has already been decoded, nothing can trigger
SendData() anymore. This patch fixes this by calling SendData() on SetPlaying().
Depends on D32111
Differential Revision: https://phabricator.services.mozilla.com/D32112
--HG--
extra : moz-landing-system : lando
If we end up taking the gap branch of the if in the patch, and the media file
in the future from the point we branched at contains no gaps or overlaps, we'll
still end up taking the gap branch until the end. This is because it only
appends up to the start of the current frame, but not the frame itself. I.e.,
when the next frame gets processed, there's still a gap in appended data.
Depends on D32107
Differential Revision: https://phabricator.services.mozilla.com/D32108
--HG--
extra : moz-landing-system : lando
We should resolve video playing promise when we finish playing the last frame. Otherwise, it would cause playback to be ended earlier than its original duration.
Differential Revision: https://phabricator.services.mozilla.com/D32051
--HG--
extra : moz-landing-system : lando
This patch structurizes the media debug information via webidl dictionaries
that are returned by HTMLMediaElement::GetMozRequestDebugInfo() and
MediaSource::GetMozDebugReaderData().
Differential Revision: https://phabricator.services.mozilla.com/D27893
--HG--
extra : moz-landing-system : lando
This moves the responsibility for creating MediaStreamTracks from
DecodedStream::Start to MediaDecoder. This let's MediaDecoder create them as
soon as metadata is known. This gives the application guarantees on when tracks
can be expected to exist, and aligns with the spec that says they should be
created when metadata is known.
Differential Revision: https://phabricator.services.mozilla.com/D28473
--HG--
extra : moz-landing-system : lando
This gives us a guarantee that the first frame of a media file can be rendered
with a second media element and mozCaptureStream(), even if the file is very
very short.
With longer video-only files there were also cases where nothing ended up being
rendered. Probably because the MediaStreamGraph ended up switching from an
AudioCallbackDriver to a SystemClockDriver and this took enough time to put the
SourceMediaStream::EndTrack and the SourceMediaStream::AddTrackListener calls
for this video track to be processed in the same iteration. The listener would
then always lose to the ending track and update main thread state too late,
leading to its media element not rendering any frames and nasty intermittent
failures.
Differential Revision: https://phabricator.services.mozilla.com/D27270
--HG--
extra : moz-landing-system : lando
A case where this wasn't working was bipbop-lateaudio.mp4, where the last frame
has duration 0 and starts and ends before the previous frame has ended. The
VideoSink still renders this frame at the end of playback, so this patch
brings DecodedStream closer to that behavior by rendering all frames with a
start time after the previous frame's start time. The track's duration is still
based on absolute times so things don't blow up.
Differential Revision: https://phabricator.services.mozilla.com/D27267
--HG--
extra : moz-landing-system : lando
This is similar to what VideoSink does, and is required for us to end playback
on HAVE_CURRENT_DATA (a frame for currentTime must be available).
Differential Revision: https://phabricator.services.mozilla.com/D27261
--HG--
extra : moz-landing-system : lando
This gives us a guarantee that the first frame of a media file can be rendered
with a second media element and mozCaptureStream(), even if the file is very
very short.
With longer video-only files there were also cases where nothing ended up being
rendered. Probably because the MediaStreamGraph ended up switching from an
AudioCallbackDriver to a SystemClockDriver and this took enough time to put the
SourceMediaStream::EndTrack and the SourceMediaStream::AddTrackListener calls
for this video track to be processed in the same iteration. The listener would
then always lose to the ending track and update main thread state too late,
leading to its media element not rendering any frames and nasty intermittent
failures.
Differential Revision: https://phabricator.services.mozilla.com/D27270
--HG--
extra : moz-landing-system : lando
A case where this wasn't working was bipbop-lateaudio.mp4, where the last frame
has duration 0 and starts and ends before the previous frame has ended. The
VideoSink still renders this frame at the end of playback, so this patch
brings DecodedStream closer to that behavior by rendering all frames with a
start time after the previous frame's start time. The track's duration is still
based on absolute times so things don't blow up.
Differential Revision: https://phabricator.services.mozilla.com/D27267
--HG--
extra : moz-landing-system : lando
This is similar to what VideoSink does, and is required for us to end playback
on HAVE_CURRENT_DATA (a frame for currentTime must be available).
Differential Revision: https://phabricator.services.mozilla.com/D27261
--HG--
extra : moz-landing-system : lando
If the input has been overflow, we don't need to calculate frames and will directly return invalid result to let caller handle the error.
Differential Revision: https://phabricator.services.mozilla.com/D27010
--HG--
extra : moz-landing-system : lando
If the input has been overflow, we don't need to calculate frames and will directly return invalid result to let caller handle the error.
Differential Revision: https://phabricator.services.mozilla.com/D27010
--HG--
extra : moz-landing-system : lando
This patch adds the number of dropped frames for each step of the process
(read/sink/compositor) and gives us more insight about where frames are
dropped, as opposed to the getVideoPlaybackQuality() API which gives the grand
total.
Differential Revision: https://phabricator.services.mozilla.com/D27488
--HG--
extra : moz-landing-system : lando
It's possible to overflow when we do the position calculation, we should only store the position which won't cause the integer overflow when adding it to start time.
Differential Revision: https://phabricator.services.mozilla.com/D25340
--HG--
extra : moz-landing-system : lando
VideoSegments still have durations, and they are still needed by the
MediaStreamGraph as it shuffles MediaSegments around.
They do not have a say in the wall-clock duration of video frames however.
Removing this should prevent any producers starting to add video chunks with
durations in the future.
Differential Revision: https://phabricator.services.mozilla.com/D22914
--HG--
extra : moz-landing-system : lando
DecodedStream has been basing video timestamps on something called
StreamTracksStartTimeStamp in MediaStreams, which call through all the way
down to the GraphDriver.
This removes the entire timestamp mechanism, except for a bit of legit
usage internally in the SystemClockDriver. Video timestamps are instead
based on the audio clock through GetPosition(), the same way the VideoSink
operates.
Differential Revision: https://phabricator.services.mozilla.com/D22896
--HG--
extra : moz-landing-system : lando
Add profiler marker to show when VideoSink sends frames to compositor
and when it discards frames that are outdated according to current
audio position.
Differential Revision: https://phabricator.services.mozilla.com/D22792
--HG--
extra : moz-landing-system : lando
mProcessedQueueLength is int32_t (wrapped around an Atomic) but it is updated from the method FramesToUsecs() which returns int64_t (wrapped around a CheckedInt) thus it is changed to unsigned int to avoid integer overflow.
Differential Revision: https://phabricator.services.mozilla.com/D22949
--HG--
extra : moz-landing-system : lando
It can be determined from the size of the buffer and the number of audio frames. Additionally, it ensures that the duration of the frame is always exactly what the AudioData contains.
Differential Revision: https://phabricator.services.mozilla.com/D20170
--HG--
extra : moz-landing-system : lando
Don't re-create a new trimmed AudioData when we want to remove some content. This remove the need for some copies.
Differential Revision: https://phabricator.services.mozilla.com/D20168
--HG--
extra : moz-landing-system : lando
This will allow to remove mFrames member and calculate from the size of the content, which will dynamically change depending on a cropping filter.
Differential Revision: https://phabricator.services.mozilla.com/D20165
--HG--
extra : moz-landing-system : lando
And we add some strong assertions that we never read passed the end of the buffer.
Differential Revision: https://phabricator.services.mozilla.com/D20162
--HG--
extra : moz-landing-system : lando
Avoid resetting volume in every AudioSink restart to stop unexpected volume change during play/pause and seek. Volume changes could occur if the volume has been modified outside firefox (for example pavucontrol in Linux).
Differential Revision: https://phabricator.services.mozilla.com/D15210
--HG--
extra : moz-landing-system : lando
The VideoSink shares the AudioSink's own EndedPromise to notify its user that it has ended. As such, the MozPromise used must be non-exclusive.
Using the GenericPromise for such purpose only hid that requirement.
We also remove the MediaSink from the media namespace, and clarify the naming of some arguments and class members to accurately describe what they do.
Differential Revision: https://phabricator.services.mozilla.com/D14024
--HG--
extra : moz-landing-system : lando
This removes DecodedStream's use of MediaStreamListener in favor of
MediaStreamTrackListener. This change has however rippled through to a lot
more cleanup, per below.
This moves the MediaStreamTrack lifetime ownership for captured
HTMLMediaElements from the media element to DecodedStream, where the
MediaStreamGraph-side tracks are already created and ended today.
This makes MediaStreamTrack creation explicit across the entire codebase and
lets us remove the MediaStreamTrackSourceGetter class and the infrastructure
of adding MediaStreamTracks after they've already been created in the graph
from DOMMediaStream.
With track ownership, and thus TrackID allocation ownership, happening
exclusively in DecodedStream for its output tracks, we also stop throwing
away and recreating the SourceMediaStream to which we feed data on seek.
This is one step closer to fixing bug 1172394 and spec compliance of
HTMLMediaElement.captureStream().
Differential Revision: https://phabricator.services.mozilla.com/D12273
--HG--
extra : moz-landing-system : lando
We only have a interest in the data which is popped out from the front side of the queue.
Differential Revision: https://phabricator.services.mozilla.com/D9190
--HG--
extra : moz-landing-system : lando
Allow AudioSinkWrapper to access MDSM's audio queue in order to know whether the audio source ended.
Differential Revision: https://phabricator.services.mozilla.com/D8031
--HG--
extra : moz-landing-system : lando
We should not init the audio steam when the audio queue is marked as finished.
Differential Revision: https://phabricator.services.mozilla.com/D8031
--HG--
extra : moz-landing-system : lando
Dorel Luca
Jim Porter
alwu
Matthew Gregan
Andreas Pehrson
Gurzau Raul
Gerald Squelart
Bogdan Tara
Jean-Yves Avenard
Narcis Beleuzu
Cosmin Sabou
Nicholas Nethercote
Tarek Ziadé
shindli
Ciure Andrei
arthur.iakab
Sylvestre Ledru
Csoregi Natalia
John Lin
Mike Conley
Alex Chronopoulos
Will Hawkins
Chris Pearce
Ehsan Akhgari
Noemi Erli
Coroiu Cristina
Daniel Varga