These early exits prevent calls to `ResourceAt(0)` on an empty queue. Such
a call emits an NS_WARN message. Both calls can either start with an empty
queue or loop until they empty the queue. Since these functions are called
frequently, the NS_WARN messages can fill up the console.
Differential Revision: https://phabricator.services.mozilla.com/D148678
After making `MediaInputPort::Get{Source, Destination}` graph-thread
only, we can no longer use them in `ConnectToNativeDevice` and
`ConnectToNonNativeDevice` since their caller run these two functions in
the main thread in our tests.
It's better to re-implement `ConnectToNativeDevice` and
`ConnectToNonNativeDevice` as main-thread APIs now.
Depends on D143780
Differential Revision: https://phabricator.services.mozilla.com/D147790
This creates a helper function in a new class gfxMacUtils that makes the
software and hardware decoders use the same logic. The new class was
necessary because gfxUtils is included in many files that don't include
CoreFoundation headers and would hit namespace collision if it was included
there.
Differential Revision: https://phabricator.services.mozilla.com/D146735
BT709 transfer characteristics may be specified in videos even if they don't
use the BT709 colorSpace. This change also makes BT709 the default, since it
is the safest option for unspecified cases.
Differential Revision: https://phabricator.services.mozilla.com/D146734
Do not use compositor device for decoder device when SW-WR + D3D11 is used.
When SW-WR + D3D11 is used, there is a case that RenderDXGITextureHost::MapPlane() is called. It readbacks video frame data from ID3D11Texture2D. Before doing the readback, copying video data/decoding video data needs to be completed. But we could not use keyed mutex for the ID3D11Texture2D.
To prevent the problem, SyncObjectD3D11Client::Synchronize() is called in D3D11DXVA2Manager::CopyToImage(). But the Synchronize() is called only when decoder device is not compositor device. Then when compositor device is not used for decoder device, the readback problem does not happen.
Differential Revision: https://phabricator.services.mozilla.com/D147049
test_mediarecorder_record_changing_video_resolution.html is very sensitive to timing,
and tsan/asan builds plus accelerated Canvas2D can mess with the timing enoug to cause
these tests to fail more frequently.
Differential Revision: https://phabricator.services.mozilla.com/D146982
The initialization itself can take some time and this audibility event is used
to prioritize the process, that will soon become audible, on at least Windows,
so it's better to get prioritized slightly ahead of time.
We should eventually untie audibility from priorities, but this hasn't happened
yet. Instead, we should make it so silent audio for an extended period of time
causes a shutdown of the cubeb stream, it's a lot more efficient.
Differential Revision: https://phabricator.services.mozilla.com/D145328
On Windows (in particular, but also not quick on other OS, depending on the
hardware being used), opening a system-level audio stream (using cubeb)
can be very long. It's a blocking call from the MDSM thread. The MDSM can't
query the clock and update video frames while the stream is being opened,
resulting in the video freezing.
On the other hand, we don't want the clock starting to increase while first
starting the media, it's better to block the MDSM in this case.
This patch changes `AudioSinkWrapper::StartAudioSink` to have two modes:
- A synchronous mode for the first opening (and therefore for seeking, etc.)
- An asynchronous mode to use when the stream is unmuted: the AudioSinkWrapper
will continue to use a clock based on the system clock, until the system-level
audio stream is effectively opened, at which point it will switch to using it
for the clock.
This new mode does the initialization on a background thread, but starts the
system-level audio stream from the MDSM thread (it's fast enough and simplifies
the code a lot).
The promises are created synchronously, because this is what the MDSM expects,
but they are rejected in case of failure.
Finally, an edge case is handled: if the stream is stopped while it's being
opened asynchronously. This is detected, and instead of starting the
system-level audio stream, the stream is shut down.
Differential Revision: https://phabricator.services.mozilla.com/D145000
No change in functionality, this is to prepare for the next patch that will do
the initialization asynchronously because it's expensing and blocking.
Differential Revision: https://phabricator.services.mozilla.com/D144999
The base clock for the new audio stream is precisely the time when the clock
source changes from the system clock to the audio clock, accounting for the time
it takes to actually starts the stream. This allows avoiding discontinuities
when switching clocks.
Differential Revision: https://phabricator.services.mozilla.com/D136237
This will allow detecting the first time the AudioSinkWrapper starts using the
clock from an audio stream after having been muted (=using the system clock) for
some time, to correct the timing.
Differential Revision: https://phabricator.services.mozilla.com/D136236
Because we're generally using high latency on the cubeb stream used by
AudioStream instance, it can take some time for the callbacks to start being
called, and the for the audio clock (cubeb_stream_get_position(...)) to advance.
This waits for the first callback to be called before using the clock of the
audio stream, and the system clock keeps being in use until then.
Differential Revision: https://phabricator.services.mozilla.com/D136235
This does the following:
- When the media is muted, shut down and release the AudioSink ;
- While the media is muted, use the system clock to make video advance ;
- Each time the clock is queried, check if some audio packets should be
discarded because they are in the past, compared to the media time ;
- While muted, if the audio finished, resolve the EndedPromise ;
- When the media is un-muted, a new AudioSink is created, and its clock starts
to be in use.
This works well and A/V sync is correct, but a micro-stuttering is perceptible
on the video when looking carefully, because of the time it takes to open the
audio stream. This is fixed in subsequent patches.
Differential Revision: https://phabricator.services.mozilla.com/D136234
We're going to shutdown the AudioStream in a subsequent patch, when audio is
muted. This will allow resolving it at the right time when the media ends while
muted.
This also extracts a method to start an AudioSink, because it's going to be
started in multiple locations in a future patch.
Differential Revision: https://phabricator.services.mozilla.com/D136233
Some of the tests in the webrtc and playback crash tests open/close lots of
stream at once, frequently putting the system audio stack in error, and no new
streams can be opened.
This patch moves a particular test that requires a working audio stack at the
beginning of the suite to make it pass consistently.
Differential Revision: https://phabricator.services.mozilla.com/D147093
Some of the tests in the webrtc and playback crash tests open/close lots of
stream at once, frequently putting the system audio stack in error, and no new
streams can be opened.
This patch moves a particular test that requires a working audio stack at the
beginning of the suite to make it pass consistently.
Differential Revision: https://phabricator.services.mozilla.com/D147093
With CI's NVIDIA GPU, SharedHandle of ID3D11Texture2D of hardware decoded video during no video copy caused rendering problem. When SharedHandle is not used, the rendering problem did not happen. But when video is rendered to WebGL texture, SharedHandle need to be used.
In this case, D3D11TextureIMFSampleImage copies original ID3D11Texture2D to a new ID3D11Texture2D and use a shared handled of the copied texture. And no video copy of future video frames are disabled.
NoCopyNV12Texture is renamed to ZeroCopyNV12Texture to clarify its meaning.
Differential Revision: https://phabricator.services.mozilla.com/D144598
This means as of this patch
- All mp4 specific audio codecs are handled. There's some more we could
theoretically get, stuff like ALAC, but it's not clear to me we handle them
following demuxing. I've left a catch all in the mp4 demuxing code just in
case.
- We no longer pack the codec-delay/preskip at the head of the opus binary blob.
This means that the binary blob is just the opus header data and the container
specific preskip has its own member. My hope is this is clearer and easier to
understand. It also means we can drop some of the code we had for packing the
delay/preskip into a binary blob.
Differential Revision: https://phabricator.services.mozilla.com/D145521
This should be the last change needed for a number of our platform specific
decoders to be switched entirely over to the new variant types. As such, we can
move them from the lenient `ForceGetAudioCodecSpecificBlob` and to
`GetAudioCodecSpecificBlob` to enforce handling of codecs that don't use
trivially blob like codec specific data.
Differential Revision: https://phabricator.services.mozilla.com/D145520
We don't store Wave specific info, but this means that we have a struct in place
if we ever want to add such data + gives us more typing info for codec specific
data.
Differential Revision: https://phabricator.services.mozilla.com/D145519
This switches the mp3 codec specific handling from using a binary blob to
instead use a typed structure.
Depends on D134729
Differential Revision: https://phabricator.services.mozilla.com/D134730
This switches existing code to use the new variant style codec specific
information for audio, but retains the binary blob style apporach everywhere.
There should be no functional changes following this refactor. Internally all
codec specific data is still stored in blobs. This allows a baseline from which
we can modify specific codecs to use more constrained versions of the variant.
Depends on D134728
Differential Revision: https://phabricator.services.mozilla.com/D134729
This introduces structures that will be used to store different audio codec
specific data. The idea is that we can communicate stricter type information
with these + better document the different structures. My primary goal is to aid
in developer comprehension, and to make changes around our codec handling easier
and less opaque.
Differential Revision: https://phabricator.services.mozilla.com/D134727
`TestDeviceInputTrack.ErrorCallback` has the same racing issue as the
bug 1765312's one (See D144637). We should wait for the
cubeb-stream-stop called by the forced-error's background thread before
calling `NonNativeInputTrack::StopAudio()` to avoid getting the stopped
state callback before the error callback.
Differential Revision: https://phabricator.services.mozilla.com/D146475
This patch won't actually build, because a few bits of code are used
for both nsIFactory::createInstance and static components, and static
components are not fixed until the next patch.
The first place is nsLoadGroupConstructor, which uses an nsIFactory
macro to create a static component constructor. (This could be worked
around by expanding the macro to the state before this patch.)
The other issue is that nsAppShellConstructor is used in an nsIFactory
on OSX, but as a static component on all other platforms. This could
be worked around by wrapping nsAppShellConstructor in an adaptor that
passes in the extra null argument to nsAppShellConstructor.
Differential Revision: https://phabricator.services.mozilla.com/D146456
A previous patch introduced an inner loop within the desktop capture run
loop. The PlatformUIThread always does a single execution of the run
loop before it goes into its alertable sleep -> dispatch -> repeat
cycle. We should not be looping within the desktop capture impl and
should allow the platform UI thread to repeated dispatch to the capture
impl instead. Additionally the startup code was racy with the shutdown
code. Fixing the former caused lock ups durring shutdown while startup
was still happening. Static analysis guards were added in this patch
to help reduce the fragility of this code.
Differential Revision: https://phabricator.services.mozilla.com/D141937
A previous patch introduced an inner loop within the desktop capture run
loop. The PlatformUIThread always does a single execution of the run
loop before it goes into its alertable sleep -> dispatch -> repeat
cycle. We should not be looping within the desktop capture impl and
should allow the platform UI thread to repeated dispatch to the capture
impl instead. Additionally the startup code was racy with the shutdown
code. Fixing the former caused lock ups durring shutdown while startup
was still happening. Static analysis guards were added in this patch
to help reduce the fragility of this code.
Differential Revision: https://phabricator.services.mozilla.com/D141937
This patch removes the redundant nsICryptoHMAC interface and implementation,
updates front-end code to use WebCrypto, and changes back-end code to use the
helper class HMAC introduced by this patch.
This also removes the last uses of nsIKeyObject and nsIKeyObjectFactory, and
thus those interfaces and implementations as well.
Differential Revision: https://phabricator.services.mozilla.com/D145656
Switch back to SW decode when frame decode time and averange decode time is bigger
than frame interval for defined number of frames.
Differential Revision: https://phabricator.services.mozilla.com/D145871
This means as of this patch
- All mp4 specific audio codecs are handled. There's some more we could
theoretically get, stuff like ALAC, but it's not clear to me we handle them
following demuxing. I've left a catch all in the mp4 demuxing code just in
case.
- We no longer pack the codec-delay/preskip at the head of the opus binary blob.
This means that the binary blob is just the opus header data and the container
specific preskip has its own member. My hope is this is clearer and easier to
understand. It also means we can drop some of the code we had for packing the
delay/preskip into a binary blob.
Differential Revision: https://phabricator.services.mozilla.com/D145521
This should be the last change needed for a number of our platform specific
decoders to be switched entirely over to the new variant types. As such, we can
move them from the lenient `ForceGetAudioCodecSpecificBlob` and to
`GetAudioCodecSpecificBlob` to enforce handling of codecs that don't use
trivially blob like codec specific data.
Differential Revision: https://phabricator.services.mozilla.com/D145520
We don't store Wave specific info, but this means that we have a struct in place
if we ever want to add such data + gives us more typing info for codec specific
data.
Differential Revision: https://phabricator.services.mozilla.com/D145519
This switches the mp3 codec specific handling from using a binary blob to
instead use a typed structure.
Depends on D134729
Differential Revision: https://phabricator.services.mozilla.com/D134730
This switches existing code to use the new variant style codec specific
information for audio, but retains the binary blob style apporach everywhere.
There should be no functional changes following this refactor. Internally all
codec specific data is still stored in blobs. This allows a baseline from which
we can modify specific codecs to use more constrained versions of the variant.
Depends on D134728
Differential Revision: https://phabricator.services.mozilla.com/D134729
This introduces structures that will be used to store different audio codec
specific data. The idea is that we can communicate stricter type information
with these + better document the different structures. My primary goal is to aid
in developer comprehension, and to make changes around our codec handling easier
and less opaque.
Differential Revision: https://phabricator.services.mozilla.com/D134727
> /builds/worker/fetches/MacOSX10.12.sdk/usr/include/MacTypes.h:542:16: error: reference to 'Point' is ambiguous
The point type conflicts with the `gfx::Point`, so move the media data out of the unify build.
Differential Revision: https://phabricator.services.mozilla.com/D146021
These state object help to store variables only be used in a certain state, which avoid the chance to modify them in an incorrect state.
Also now it's able to do the cleaning for the state transition.
Differential Revision: https://phabricator.services.mozilla.com/D143678
Now it's just used for testing and development purpose.
In the future, this should only be used for encrypted playback and we would also need a fallback mechanism if the external engine fails.
Differential Revision: https://phabricator.services.mozilla.com/D140592
Implement a new decoder module which would use the media engine Id to create the decoder, which is implemented by WMF Media Engine API.
The actual data flow would be like that, the encoded data would be passed to the RDD process by using the remote decoder, and remote decoder would feed the data into media engine decoder, such as MFMediaEngineVideoStream and MFMediaEngineAudioStream.
Differential Revision: https://phabricator.services.mozilla.com/D143805
In this patch, we assign the Media Engine Id to the format reader, and append that Id into CreateDecoderParams when we want to create a decoder.
That Id would be used as a hint in order to find a correct decoder to send the data to the remote media engine in following patches.
Depends on D139204
Differential Revision: https://phabricator.services.mozilla.com/D140153
This patch implements a new IPDL which is used for communicating with the remote media engine. It will be used between the content process and the RDD process.
For each media engine pair, they would share an unique ID, which is used to indentify different engine pairs (if any) and for querying a specific engine via Id, which is implemented in the next patch.
Depends on D140014
Differential Revision: https://phabricator.services.mozilla.com/D139204
This patch introduces a new type of state machine that is used when the playback is controlled by an external playback engine which would provide high level operations but hide most low level tasks from its client. Eg. Media Foundation Media Engine.
The extenral engine should use events to communicate this state machine, and state machine should address those events in order to fullfill the need of the external engine.
Differential Revision: https://phabricator.services.mozilla.com/D140014
In following patch, we will implement a new type of state machine. In order to avoid redundant code, this patch splits some basic share codes into a new base class.
Differential Revision: https://phabricator.services.mozilla.com/D140013
> /builds/worker/fetches/MacOSX10.12.sdk/usr/include/MacTypes.h:542:16: error: reference to 'Point' is ambiguous
The point type conflicts with the `gfx::Point`, so move the media data out of the unify build.
Differential Revision: https://phabricator.services.mozilla.com/D146021
These state object help to store variables only be used in a certain state, which avoid the chance to modify them in an incorrect state.
Also now it's able to do the cleaning for the state transition.
Differential Revision: https://phabricator.services.mozilla.com/D143678
Now it's just used for testing and development purpose.
In the future, this should only be used for encrypted playback and we would also need a fallback mechanism if the external engine fails.
Differential Revision: https://phabricator.services.mozilla.com/D140592
Implement a new decoder module which would use the media engine Id to create the decoder, which is implemented by WMF Media Engine API.
The actual data flow would be like that, the encoded data would be passed to the RDD process by using the remote decoder, and remote decoder would feed the data into media engine decoder, such as MFMediaEngineVideoStream and MFMediaEngineAudioStream.
Differential Revision: https://phabricator.services.mozilla.com/D143805
In this patch, we assign the Media Engine Id to the format reader, and append that Id into CreateDecoderParams when we want to create a decoder.
That Id would be used as a hint in order to find a correct decoder to send the data to the remote media engine in following patches.
Depends on D139204
Differential Revision: https://phabricator.services.mozilla.com/D140153
This patch implements a new IPDL which is used for communicating with the remote media engine. It will be used between the content process and the RDD process.
For each media engine pair, they would share an unique ID, which is used to indentify different engine pairs (if any) and for querying a specific engine via Id, which is implemented in the next patch.
Depends on D140014
Differential Revision: https://phabricator.services.mozilla.com/D139204
This patch introduces a new type of state machine that is used when the playback is controlled by an external playback engine which would provide high level operations but hide most low level tasks from its client. Eg. Media Foundation Media Engine.
The extenral engine should use events to communicate this state machine, and state machine should address those events in order to fullfill the need of the external engine.
Differential Revision: https://phabricator.services.mozilla.com/D140014
In following patch, we will implement a new type of state machine. In order to avoid redundant code, this patch splits some basic share codes into a new base class.
Differential Revision: https://phabricator.services.mozilla.com/D140013
The initialization itself can take some time and this audibility event is used
to prioritize the process, that will soon become audible, on at least Windows,
so it's better to get prioritized slightly ahead of time.
We should eventually untie audibility from priorities, but this hasn't happened
yet. Instead, we should make it so silent audio for an extended period of time
causes a shutdown of the cubeb stream, it's a lot more efficient.
Differential Revision: https://phabricator.services.mozilla.com/D145328
On Windows (in particular, but also not quick on other OS, depending on the
hardware being used), opening a system-level audio stream (using cubeb)
can be very long. It's a blocking call from the MDSM thread. The MDSM can't
query the clock and update video frames while the stream is being opened,
resulting in the video freezing.
On the other hand, we don't want the clock starting to increase while first
starting the media, it's better to block the MDSM in this case.
This patch changes `AudioSinkWrapper::StartAudioSink` to have two modes:
- A synchronous mode for the first opening (and therefore for seeking, etc.)
- An asynchronous mode to use when the stream is unmuted: the AudioSinkWrapper
will continue to use a clock based on the system clock, until the system-level
audio stream is effectively opened, at which point it will switch to using it
for the clock.
This new mode does the initialization on a background thread, but starts the
system-level audio stream from the MDSM thread (it's fast enough and simplifies
the code a lot).
The promises are created synchronously, because this is what the MDSM expects,
but they are rejected in case of failure.
Finally, an edge case is handled: if the stream is stopped while it's being
opened asynchronously. This is detected, and instead of starting the
system-level audio stream, the stream is shut down.
Differential Revision: https://phabricator.services.mozilla.com/D145000
No change in functionality, this is to prepare for the next patch that will do
the initialization asynchronously because it's expensing and blocking.
Differential Revision: https://phabricator.services.mozilla.com/D144999
The base clock for the new audio stream is precisely the time when the clock
source changes from the system clock to the audio clock, accounting for the time
it takes to actually starts the stream. This allows avoiding discontinuities
when switching clocks.
Differential Revision: https://phabricator.services.mozilla.com/D136237
This will allow detecting the first time the AudioSinkWrapper starts using the
clock from an audio stream after having been muted (=using the system clock) for
some time, to correct the timing.
Differential Revision: https://phabricator.services.mozilla.com/D136236
Because we're generally using high latency on the cubeb stream used by
AudioStream instance, it can take some time for the callbacks to start being
called, and the for the audio clock (cubeb_stream_get_position(...)) to advance.
This waits for the first callback to be called before using the clock of the
audio stream, and the system clock keeps being in use until then.
Differential Revision: https://phabricator.services.mozilla.com/D136235
This does the following:
- When the media is muted, shut down and release the AudioSink ;
- While the media is muted, use the system clock to make video advance ;
- Each time the clock is queried, check if some audio packets should be
discarded because they are in the past, compared to the media time ;
- While muted, if the audio finished, resolve the EndedPromise ;
- When the media is un-muted, a new AudioSink is created, and its clock starts
to be in use.
This works well and A/V sync is correct, but a micro-stuttering is perceptible
on the video when looking carefully, because of the time it takes to open the
audio stream. This is fixed in subsequent patches.
Differential Revision: https://phabricator.services.mozilla.com/D136234
We're going to shutdown the AudioStream in a subsequent patch, when audio is
muted. This will allow resolving it at the right time when the media ends while
muted.
This also extracts a method to start an AudioSink, because it's going to be
started in multiple locations in a future patch.
Differential Revision: https://phabricator.services.mozilla.com/D136233
Crashes of Bug 1764753 seemed to happen during closing video MFTDecoder or closing IMFSample after closing video MFTDecoder. We might need to release all IMFSamples before destroying video MFTDecoder.
Differential Revision: https://phabricator.services.mozilla.com/D144537
This one reverts Bug 1759137 and switch back to reference hw_frames_ctx. We need that as hw_frames_ctx holds surface pool which can be recreated when h.264 VA-API decoder seeks in video stream.
Differential Revision: https://phabricator.services.mozilla.com/D145094
HasPluginForAPI in the parent process doesn't really work properly
anymore. Instead just check for the API and tags when building the
capability list.
Differential Revision: https://phabricator.services.mozilla.com/D145474
test_capture_stream_av_sync is still intermittent when running under asan,
even though it was disabled for tsan in bug 1713397. This gets much worse
then run under accelerated Canvas2D, so just disable it for asan for now
as well to solve the intermittent.
Differential Revision: https://phabricator.services.mozilla.com/D144981
dom/bindings/BindingUtils.cpp(202,62): error: passing object of class type 'typename raw_type<char16_t, int>::type' (aka 'char16ptr_t') through variadic function [-Werror,-Wclass-varargs]
static_cast<unsigned>(errorNumber), funcNameStr.get(),
^
dom/bindings/BindingUtils.cpp(203,26): error: passing object of class type 'typename raw_type<char16_t, int>::type' (aka 'char16ptr_t') through variadic function [-Werror,-Wclass-varargs]
ifaceName.get());
^
dom/media/webrtc/transport/third_party/nICEr/src/ice/ice_component.c(582,15): error: passing object of class type 'nr_transport_addr' (aka 'struct nr_transport_addr_') through variadic function [-Werror,-Wclass-varargs]
component->stream->turn_servers[j].turn_server.addr);
^
toolkit/xre/dllservices/tests/gtest/TestUntrustedModules.cpp(44,45): error: passing object of class type 'typename raw_type<char16_t, int>::type' (aka 'char16ptr_t') through variadic function [-Werror,-Wclass-varargs]
wprintf(L"%s is not registered.\n", aNames[i].get());
^
toolkit/xre/dllservices/tests/gtest/TestUntrustedModules.cpp(49,30): error: passing object of class type 'typename raw_type<char16_t, int>::type' (aka 'char16ptr_t') through variadic function [-Werror,-Wclass-varargs]
wprintf(L"%s:%4d\n", aNames[i].get(), *entry);
^
toolkit/xre/dllservices/tests/gtest/TestUntrustedModules.cpp(248,30): error: passing object of class type 'typename raw_type<char16_t, int>::type' (aka 'char16ptr_t') through variadic function [-Werror,-Wclass-varargs]
wprintf(L"JSON: %s\n", json.get());
^
xpcom/io/nsLocalFileWin.cpp(1647,20): error: passing object of class type 'typename raw_type<char16_t, int>::type' (aka 'char16ptr_t') through variadic function [-Werror,-Wclass-varargs]
NS_ConvertASCIItoUTF16(nsDependentCString(aField)).get());
^
Differential Revision: https://phabricator.services.mozilla.com/D144665
Brad Werth
Cosmin Sabou
Michael Froman
alwu
Mike Hommey
Jan-Ivar Bruaroey
Mark Banner
Sylvestre Ledru
Tooru Fujisawa
serge-sans-paille
Kagami Sascha Rosylight
Paul Adenot
Tom Ritter
Chun-Min Chang
Marian-Vasile Laza
Serge Guelton
Iulian Moraru
Alexandre Lissy
Jeff Muizelaar
Masayuki Nakano
Alexandre Lissy
stransky
sotaro
Lee Salzman
Molnar Sandor
Bogdan Szekely
Bryce Seager van Dyk
Joel Maher
Andrew McCreight
Sebastian Hengst
Nico Grunbaum
Byron Campen
Jens Stutte
Dana Keeler
Randell Jesup
criss
Cristian Tuns
Andreea Pavel
Butkovits Atila
Narcis Beleuzu
Norisz Fay
Chris H-C
Karl Tomlinson
Zaggy1024