In this patch, we move the responsibility of accumulating time and report the telemetry to `TelemetryProbesReporter`.
There are some differences between new telemetry report and the old one.
1. more accuracy on knowing if element is visible
2. more accuracy on determining when it should start accumulating visible & invisible play time
3. being able to report the correct result when element encounts an error or changes to a new resource
4. report result whenever MediaDecoder stops working
Here is the explanation [1] describing why our previous method was not able to achieve those advantages.
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1685399#c13
Differential Revision: https://phabricator.services.mozilla.com/D101112
There is no need for decoder to use both "document visibility" and "element's layout visibility" to determine if we should suspend decoding.
That can simply be done by checking `HTMLMediaElement::IsActuallyInvisible()`.
Differential Revision: https://phabricator.services.mozilla.com/D101108
When enabling our MediaTrack implementation (which we don't plan to by default,
NB) and disabling all audio tracks and unselecting all video tracks while having
an active captureStream leads to having no output tracks in DecodedStream.
In this case, DecodedStream doesn't know which graph to use for creating the
intermediary tracks it feeds data to. We don't want to resort to the default
graph either, since two graphs on different clocks could then race each other.
With this patch we plumb down a SharedDummyTrack from the media element where
the captureStream was triggered, through MediaDecoder, to DecodedStream. The
SharedDummyTrack guarantees to keep the graph alive, and holds the graph used
for the output tracks.
Differential Revision: https://phabricator.services.mozilla.com/D99822
This patch was generated by running:
```
perl -p -i \
-e 's/^(\s+)([a-zA-Z0-9.]+) = NS_ConvertUTF8toUTF16\((.*)\);/\1CopyUTF8toUTF16(\3, \2);/;' \
-e 's/^(\s+)([a-zA-Z0-9.]+) = NS_ConvertUTF16toUTF8\((.*)\);/\1CopyUTF16toUTF8(\3, \2);/;' \
$FILE
```
against every .cpp and .h in mozilla-central, and then fixing up the
inevitable errors that happen as a result of matching C++ expressions with
regexes. The errors fell into three categories:
1. Calling the convert functions with `std::string::c_str()`; these were
changed to simply pass the string instead, relying on implicit conversion
to `mozilla::Span`.
2. Calling the convert functions with raw pointers, which is not permitted
with the copy functions; these were changed to invoke `MakeStringSpan` first.
3. Other miscellaneous errors resulting from over-eager regexes and/or the
replacement not being type-aware. These changes were reverted.
Differential Revision: https://phabricator.services.mozilla.com/D88903
AutoEnter was an attempt around a race between AbstractThread and MessageLoopAbstractThreadWrap that would cause AbstractThread::GetCurrent() to return an incorrect value. MessageLoopAbstractThreadWrapper is no more and as such AutoEnter is no longer required.
Differential Revision: https://phabricator.services.mozilla.com/D71279
Implement the `MediaDecoder::Suspend/Resume` to shutdown and recreate the MediaSink. This is important the page is refreshed because the existing tree does not shutdown immediatelly. It stays alive but remains suspended thus it is necessary to clean up the low-level system resources. In order to support the suspended state with the existing work flow, if MediaSink is suspended it is disallowed to start it or change the sink id. Recreating the MediaSink is possible but starting it is restricted until resume.
Differential Revision: https://phabricator.services.mozilla.com/D63831
--HG--
extra : moz-landing-system : lando
This converts `nsAutoPtr` usage in dom/media to `UniquePtr`. Beyond just a
search and replace we also needed to update assignment and access of the
`UniquePtr`s. This falls into a few categories:
- Assignment from a newly constructed object switches to `MakeUnique`
- Assignment from a raw ptr switches to `UniquePtr::reset`
- Handing out a raw ptr now requires `UniquePtr::get`
- Uses `UniquePtr::release` rather than `nsAutoPtr::forget`
- A few spots are updated to return a `UniquePtr` rather than a raw ptr
Differential Revision: https://phabricator.services.mozilla.com/D60084
--HG--
extra : moz-landing-system : lando
When media element is inactive, asking MDSM to seek is actually useless, because all the data we decode won't be showed to user. In addition, we have to store extra pending events for `seeking` and `seeked`, which might result in memory overflow if the inactive page is calling `seek()` all the time.
Therfore, we should delay all seeking tasks while media is inactive, and perform the latest seeking task when media becomes active.
Differential Revision: https://phabricator.services.mozilla.com/D58918
--HG--
extra : moz-landing-system : lando
The inclusions were removed with the following very crude script and the
resulting breakage was fixed up by hand. The manual fixups did either
revert the changes done by the script, replace a generic header with a more
specific one or replace a header with a forward declaration.
find . -name "*.idl" | grep -v web-platform | grep -v third_party | while read path; do
interfaces=$(grep "^\(class\|interface\).*:.*" "$path" | cut -d' ' -f2)
if [ -n "$interfaces" ]; then
if [[ "$interfaces" == *$'\n'* ]]; then
regexp="\("
for i in $interfaces; do regexp="$regexp$i\|"; done
regexp="${regexp%%\\\|}\)"
else
regexp="$interfaces"
fi
interface=$(basename "$path")
rg -l "#include.*${interface%%.idl}.h" . | while read path2; do
hits=$(grep -v "#include.*${interface%%.idl}.h" "$path2" | grep -c "$regexp" )
if [ $hits -eq 0 ]; then
echo "Removing ${interface} from ${path2}"
grep -v "#include.*${interface%%.idl}.h" "$path2" > "$path2".tmp
mv -f "$path2".tmp "$path2"
fi
done
fi
done
Differential Revision: https://phabricator.services.mozilla.com/D55442
--HG--
extra : moz-landing-system : lando
The inclusions were removed with the following very crude script and the
resulting breakage was fixed up by hand. The manual fixups did either
revert the changes done by the script, replace a generic header with a more
specific one or replace a header with a forward declaration.
find . -name "*.idl" | grep -v web-platform | grep -v third_party | while read path; do
interfaces=$(grep "^\(class\|interface\).*:.*" "$path" | cut -d' ' -f2)
if [ -n "$interfaces" ]; then
if [[ "$interfaces" == *$'\n'* ]]; then
regexp="\("
for i in $interfaces; do regexp="$regexp$i\|"; done
regexp="${regexp%%\\\|}\)"
else
regexp="$interfaces"
fi
interface=$(basename "$path")
rg -l "#include.*${interface%%.idl}.h" . | while read path2; do
hits=$(grep -v "#include.*${interface%%.idl}.h" "$path2" | grep -c "$regexp" )
if [ $hits -eq 0 ]; then
echo "Removing ${interface} from ${path2}"
grep -v "#include.*${interface%%.idl}.h" "$path2" > "$path2".tmp
mv -f "$path2".tmp "$path2"
fi
done
fi
done
Differential Revision: https://phabricator.services.mozilla.com/D55442
--HG--
extra : moz-landing-system : lando
This patch does the following:
- Makes cloneElementVisually() return a promise
- Plumbs an event from the MediaDecoderStateMachine's VideoSink to
HTMLVideoElement
- Hooks the event up to resolve the promise from cloneElementVisually()
- Updates tests and their expectations.
Differential Revision: https://phabricator.services.mozilla.com/D53831
--HG--
extra : moz-landing-system : lando
This better follows how MediaDecoder and MediaDecoderStateMachine was
architected.
Differential Revision: https://phabricator.services.mozilla.com/D53710
--HG--
extra : moz-landing-system : lando
This patch does the following:
- Makes cloneElementVisually() return a promise
- Plumbs an event from the MediaDecoderStateMachine's VideoSink to
HTMLVideoElement
- Hooks the event up to resolve the promise from cloneElementVisually()
- Updates tests and their expectations.
Differential Revision: https://phabricator.services.mozilla.com/D53831
--HG--
extra : moz-landing-system : lando
This better follows how MediaDecoder and MediaDecoderStateMachine was
architected.
Differential Revision: https://phabricator.services.mozilla.com/D53710
--HG--
extra : moz-landing-system : lando
This patches does several minor things:
- Moves SetSink (from setSinkid) to automatic coalescing of multiple calls
through a Canonical/Mirror setup instead of a manual atomic counter.
- Simplifies the logic for when to update the sink in SetSink.
- Removes PlaybackParams as a general MediaSink property, as it only contains
audio params.
- Makes PlaybackParams an internal AudioSink concept, that AudioSinkWrapper
knows about.
- Ensures mMediaSink is only accessed on the decoder TaskQueue, to allow
accessing mirrored members when creating it.
Differential Revision: https://phabricator.services.mozilla.com/D52043
--HG--
extra : moz-landing-system : lando
This patch removes the responsibility of js-facing MediaStreamTracks from the
MediaDecoder stack, and moves the machinery for setting up DecodedStream to
higher order functions like state mirroring and watchables.
OutputStreamManager is completely gone, since it was designed to manage
MediaStreamTracks across multiple output streams for a single decoder,
on main thread. HTMLMediaElement took over its task in the previous patch.
The MediaDecoderStateMachine now has three control points for capturing:
- mOutputCaptured, which, if true, will capture all decoded data into
mOutputTracks. If this is set, but mOutputTracks is empty, we are still
waiting for tracks, and DecodedStream will not play any data. When tracks are
set, a new DecodedStream is created that will play data through
SourceMediaTracks piped into mOutputTracks.
- mOutputTracks, which is the set of tracks data is captured into, for
forwarding to all the output tracks the media element is managing. This set of
tracks is managed by the MediaDecoder owner, and must contain one audio track
if the decoder is decoding audio, and one video track if the decoder is
decoding video. It may be empty since output can be captured before metadata
is loaded, or playback has ended.
- mOutputPrincipal, which is the principal of the decoded data. All data sent
into SourceMediaTracks is tagged with this principal.
Differential Revision: https://phabricator.services.mozilla.com/D52042
--HG--
extra : moz-landing-system : lando
This reworks how media element captureStream works by removing the differences
between MediaStream and MediaDecoder capture. MediaDecoder capture will be
refactored so that ownership of MediaStreamTracks lies with the media element
instead of the OutputStreamManager. The internal MediaDecoder parts happen in a
later patch.
The new API for capturing a MediaDecoder involves a boolean on/off toggle, the
output tracks the decoder pipes data to, and the principal that data is tagged
with. If capturing is on but there are no output tracks, playback will not
happen, to ensure that no data gets accidentally skipped in the output tracks
while captured.
This also changes the logic for setting up MediaElementTrackSources in
HTMLMediaElement so it's triggered by the WatchManager and thus run in tail
dispatched runnables.
Differential Revision: https://phabricator.services.mozilla.com/D52040
--HG--
extra : moz-landing-system : lando
HTMLMediaElement avoid creating new tracks in MetadataLoaded when it has already
created some, so there should be no side effect to this patch.
Differential Revision: https://phabricator.services.mozilla.com/D52037
--HG--
extra : moz-landing-system : lando
This patches does several minor things:
- Moves SetSink (from setSinkid) to automatic coalescing of multiple calls
through a Canonical/Mirror setup instead of a manual atomic counter.
- Simplifies the logic for when to update the sink in SetSink.
- Removes PlaybackParams as a general MediaSink property, as it only contains
audio params.
- Makes PlaybackParams an internal AudioSink concept, that AudioSinkWrapper
knows about.
- Ensures mMediaSink is only accessed on the decoder TaskQueue, to allow
accessing mirrored members when creating it.
Differential Revision: https://phabricator.services.mozilla.com/D52043
--HG--
extra : moz-landing-system : lando
This patch removes the responsibility of js-facing MediaStreamTracks from the
MediaDecoder stack, and moves the machinery for setting up DecodedStream to
higher order functions like state mirroring and watchables.
OutputStreamManager is completely gone, since it was designed to manage
MediaStreamTracks across multiple output streams for a single decoder,
on main thread. HTMLMediaElement took over its task in the previous patch.
The MediaDecoderStateMachine now has three control points for capturing:
- mOutputCaptured, which, if true, will capture all decoded data into
mOutputTracks. If this is set, but mOutputTracks is empty, we are still
waiting for tracks, and DecodedStream will not play any data. When tracks are
set, a new DecodedStream is created that will play data through
SourceMediaTracks piped into mOutputTracks.
- mOutputTracks, which is the set of tracks data is captured into, for
forwarding to all the output tracks the media element is managing. This set of
tracks is managed by the MediaDecoder owner, and must contain one audio track
if the decoder is decoding audio, and one video track if the decoder is
decoding video. It may be empty since output can be captured before metadata
is loaded, or playback has ended.
- mOutputPrincipal, which is the principal of the decoded data. All data sent
into SourceMediaTracks is tagged with this principal.
Differential Revision: https://phabricator.services.mozilla.com/D52042
--HG--
extra : moz-landing-system : lando
This reworks how media element captureStream works by removing the differences
between MediaStream and MediaDecoder capture. MediaDecoder capture will be
refactored so that ownership of MediaStreamTracks lies with the media element
instead of the OutputStreamManager. The internal MediaDecoder parts happen in a
later patch.
The new API for capturing a MediaDecoder involves a boolean on/off toggle, the
output tracks the decoder pipes data to, and the principal that data is tagged
with. If capturing is on but there are no output tracks, playback will not
happen, to ensure that no data gets accidentally skipped in the output tracks
while captured.
This also changes the logic for setting up MediaElementTrackSources in
HTMLMediaElement so it's triggered by the WatchManager and thus run in tail
dispatched runnables.
Differential Revision: https://phabricator.services.mozilla.com/D52040
--HG--
extra : moz-landing-system : lando
HTMLMediaElement avoid creating new tracks in MetadataLoaded when it has already
created some, so there should be no side effect to this patch.
Differential Revision: https://phabricator.services.mozilla.com/D52037
--HG--
extra : moz-landing-system : lando
Brendan Early
alwu
Robert Bryer
Bogdan Tara
Andreas Pehrson
Simon Giesecke
Andi-Bogdan Postelnicu
Nathan Froyd
Jean-Yves Avenard
Andreas Farre
Alex Chronopoulos
Eric Rahm
Gabriele Svelto
Dorel Luca
Daniel Varga
Csoregi Natalia
Ryan VanderMeulen
Gurzau Raul
Karl Tomlinson