The initialization itself can take some time and this audibility event is used
to prioritize the process, that will soon become audible, on at least Windows,
so it's better to get prioritized slightly ahead of time.
We should eventually untie audibility from priorities, but this hasn't happened
yet. Instead, we should make it so silent audio for an extended period of time
causes a shutdown of the cubeb stream, it's a lot more efficient.
Differential Revision: https://phabricator.services.mozilla.com/D145328
On Windows (in particular, but also not quick on other OS, depending on the
hardware being used), opening a system-level audio stream (using cubeb)
can be very long. It's a blocking call from the MDSM thread. The MDSM can't
query the clock and update video frames while the stream is being opened,
resulting in the video freezing.
On the other hand, we don't want the clock starting to increase while first
starting the media, it's better to block the MDSM in this case.
This patch changes `AudioSinkWrapper::StartAudioSink` to have two modes:
- A synchronous mode for the first opening (and therefore for seeking, etc.)
- An asynchronous mode to use when the stream is unmuted: the AudioSinkWrapper
will continue to use a clock based on the system clock, until the system-level
audio stream is effectively opened, at which point it will switch to using it
for the clock.
This new mode does the initialization on a background thread, but starts the
system-level audio stream from the MDSM thread (it's fast enough and simplifies
the code a lot).
The promises are created synchronously, because this is what the MDSM expects,
but they are rejected in case of failure.
Finally, an edge case is handled: if the stream is stopped while it's being
opened asynchronously. This is detected, and instead of starting the
system-level audio stream, the stream is shut down.
Differential Revision: https://phabricator.services.mozilla.com/D145000
No change in functionality, this is to prepare for the next patch that will do
the initialization asynchronously because it's expensing and blocking.
Differential Revision: https://phabricator.services.mozilla.com/D144999
The base clock for the new audio stream is precisely the time when the clock
source changes from the system clock to the audio clock, accounting for the time
it takes to actually starts the stream. This allows avoiding discontinuities
when switching clocks.
Differential Revision: https://phabricator.services.mozilla.com/D136237
This will allow detecting the first time the AudioSinkWrapper starts using the
clock from an audio stream after having been muted (=using the system clock) for
some time, to correct the timing.
Differential Revision: https://phabricator.services.mozilla.com/D136236
Because we're generally using high latency on the cubeb stream used by
AudioStream instance, it can take some time for the callbacks to start being
called, and the for the audio clock (cubeb_stream_get_position(...)) to advance.
This waits for the first callback to be called before using the clock of the
audio stream, and the system clock keeps being in use until then.
Differential Revision: https://phabricator.services.mozilla.com/D136235
This does the following:
- When the media is muted, shut down and release the AudioSink ;
- While the media is muted, use the system clock to make video advance ;
- Each time the clock is queried, check if some audio packets should be
discarded because they are in the past, compared to the media time ;
- While muted, if the audio finished, resolve the EndedPromise ;
- When the media is un-muted, a new AudioSink is created, and its clock starts
to be in use.
This works well and A/V sync is correct, but a micro-stuttering is perceptible
on the video when looking carefully, because of the time it takes to open the
audio stream. This is fixed in subsequent patches.
Differential Revision: https://phabricator.services.mozilla.com/D136234
We're going to shutdown the AudioStream in a subsequent patch, when audio is
muted. This will allow resolving it at the right time when the media ends while
muted.
This also extracts a method to start an AudioSink, because it's going to be
started in multiple locations in a future patch.
Differential Revision: https://phabricator.services.mozilla.com/D136233
The initialization itself can take some time and this audibility event is used
to prioritize the process, that will soon become audible, on at least Windows,
so it's better to get prioritized slightly ahead of time.
We should eventually untie audibility from priorities, but this hasn't happened
yet. Instead, we should make it so silent audio for an extended period of time
causes a shutdown of the cubeb stream, it's a lot more efficient.
Differential Revision: https://phabricator.services.mozilla.com/D145328
On Windows (in particular, but also not quick on other OS, depending on the
hardware being used), opening a system-level audio stream (using cubeb)
can be very long. It's a blocking call from the MDSM thread. The MDSM can't
query the clock and update video frames while the stream is being opened,
resulting in the video freezing.
On the other hand, we don't want the clock starting to increase while first
starting the media, it's better to block the MDSM in this case.
This patch changes `AudioSinkWrapper::StartAudioSink` to have two modes:
- A synchronous mode for the first opening (and therefore for seeking, etc.)
- An asynchronous mode to use when the stream is unmuted: the AudioSinkWrapper
will continue to use a clock based on the system clock, until the system-level
audio stream is effectively opened, at which point it will switch to using it
for the clock.
This new mode does the initialization on a background thread, but starts the
system-level audio stream from the MDSM thread (it's fast enough and simplifies
the code a lot).
The promises are created synchronously, because this is what the MDSM expects,
but they are rejected in case of failure.
Finally, an edge case is handled: if the stream is stopped while it's being
opened asynchronously. This is detected, and instead of starting the
system-level audio stream, the stream is shut down.
Differential Revision: https://phabricator.services.mozilla.com/D145000
No change in functionality, this is to prepare for the next patch that will do
the initialization asynchronously because it's expensing and blocking.
Differential Revision: https://phabricator.services.mozilla.com/D144999
The base clock for the new audio stream is precisely the time when the clock
source changes from the system clock to the audio clock, accounting for the time
it takes to actually starts the stream. This allows avoiding discontinuities
when switching clocks.
Differential Revision: https://phabricator.services.mozilla.com/D136237
This will allow detecting the first time the AudioSinkWrapper starts using the
clock from an audio stream after having been muted (=using the system clock) for
some time, to correct the timing.
Differential Revision: https://phabricator.services.mozilla.com/D136236
Because we're generally using high latency on the cubeb stream used by
AudioStream instance, it can take some time for the callbacks to start being
called, and the for the audio clock (cubeb_stream_get_position(...)) to advance.
This waits for the first callback to be called before using the clock of the
audio stream, and the system clock keeps being in use until then.
Differential Revision: https://phabricator.services.mozilla.com/D136235
This does the following:
- When the media is muted, shut down and release the AudioSink ;
- While the media is muted, use the system clock to make video advance ;
- Each time the clock is queried, check if some audio packets should be
discarded because they are in the past, compared to the media time ;
- While muted, if the audio finished, resolve the EndedPromise ;
- When the media is un-muted, a new AudioSink is created, and its clock starts
to be in use.
This works well and A/V sync is correct, but a micro-stuttering is perceptible
on the video when looking carefully, because of the time it takes to open the
audio stream. This is fixed in subsequent patches.
Differential Revision: https://phabricator.services.mozilla.com/D136234
We're going to shutdown the AudioStream in a subsequent patch, when audio is
muted. This will allow resolving it at the right time when the media ends while
muted.
This also extracts a method to start an AudioSink, because it's going to be
started in multiple locations in a future patch.
Differential Revision: https://phabricator.services.mozilla.com/D136233
The base clock for the new audio stream is precisely the time when the clock
source changes from the system clock to the audio clock, accounting for the time
it takes to actually starts the stream. This allows avoiding discontinuities
when switching clocks.
Differential Revision: https://phabricator.services.mozilla.com/D136237
This will allow detecting the first time the AudioSinkWrapper starts using the
clock from an audio stream after having been muted (=using the system clock) for
some time, to correct the timing.
Differential Revision: https://phabricator.services.mozilla.com/D136236
Because we're generally using high latency on the cubeb stream used by
AudioStream instance, it can take some time for the callbacks to start being
called, and the for the audio clock (cubeb_stream_get_position(...)) to advance.
This waits for the first callback to be called before using the clock of the
audio stream, and the system clock keeps being in use until then.
Differential Revision: https://phabricator.services.mozilla.com/D136235
This does the following:
- When the media is muted, shut down and release the AudioSink ;
- While the media is muted, use the system clock to make video advance ;
- Each time the clock is queried, check if some audio packets should be
discarded because they are in the past, compared to the media time ;
- While muted, if the audio finished, resolve the EndedPromise ;
- When the media is un-muted, a new AudioSink is created, and its clock starts
to be in use.
This works well and A/V sync is correct, but a micro-stuttering is perceptible
on the video when looking carefully, because of the time it takes to open the
audio stream. This is fixed in subsequent patches.
Differential Revision: https://phabricator.services.mozilla.com/D136234
We're going to shutdown the AudioStream in a subsequent patch, when audio is
muted. This will allow resolving it at the right time when the media ends while
muted.
This also extracts a method to start an AudioSink, because it's going to be
started in multiple locations in a future patch.
Differential Revision: https://phabricator.services.mozilla.com/D136233
Before this change, the converter would be drained after each packet would be
pushed to the queue, introducing discontinuities in the audio, at the end, after
all the decoding operations were finished, but before all the audio packets were
pushed to the queue.
This went unnoticed because there is no converter most of the time, with audio
that has a sample-rate higher or equal to the common 44100Hz value.
Depends on D141356
Differential Revision: https://phabricator.services.mozilla.com/D141357
Marian-Vasile Laza
Paul Adenot
Norisz Fay
Cristian Tuns
Jeff Muizelaar
Randell Jesup
Lee Salzman
smolnar