The old technique was slightly misleading. It was counting the audio already
sent to the device, in the decoded audio duration. But in fact, there's nothing
that can be done about this, the audio is going to be played regardless of what
happens. If the audio latency is high enough, this would return a number that's
high, but not a lot of audio would be sitting in Firefox's buffer, ready to
service the audio callbacks.
It's best to only consider the decoded audio (processed + unprocessed) here,
since the goal of this is to determine whether more audio needs to be decoded.
It also saves a `cubeb_stream_get_position` IPC call if using AudioIPC (soon to
be all platforms).
Differential Revision: https://phabricator.services.mozilla.com/D138318
The queue directly contains the audio frames, not packets. We'll be able to tune
its size later. `AudioSink::PopFrames` pull from the queue, and is called in the
audio callback. The decoded frame refill logic is adjusted to be simpler.
Signaling the event to refill still takes a lock, that's going to be fixed later.
Ended(), the second member of the interface, is switched to using an atomic.
After this patch, the audio callback doesn't need to take the AudioStream lock.
Differential Revision: https://phabricator.services.mozilla.com/D137303
-Wshadow warnings are not enabled globally, so these -Wno-shadow suppressions have no effect. I had intended to enable -Wshadow globally along with these suppressions in some directories (in bug 1272513), but that was blocked by other issues.
There are too many -Wshadow warnings (now over 2000) to realistically fix them all. We should remove all these unnecessary -Wno-shadow flags cluttering many moz.build files.
Differential Revision: https://phabricator.services.mozilla.com/D132289
`profiler_thread_is_being_profiled` is used a lot for markers, so it makes sense to have a specialized version, which is a bit shorter, and lives in ProfilerMarkers.h.
Differential Revision: https://phabricator.services.mozilla.com/D130009
`profiler_thread_is_being_profiled` is used a lot for markers, so it makes sense to have a specialized version, which is a bit shorter, and lives in ProfilerMarkers.h.
Differential Revision: https://phabricator.services.mozilla.com/D130009
When we adjust the playback rate on the audio track, the audio clock time would be no longer align with the graph time.
Eg. playback rate=2, when the graph time passes 10s, the audio clock time actually already goes forward 20s.
After audio track ended, the video track would start to drive the clock time but the video clock time is align with the graph time, which means it would be smaller than the audio clock in that situation.
Therefore, we have to ignore the video clock time in that case. In addition, this patch also address the duration change on video frames based on the playback rate.
Differential Revision: https://phabricator.services.mozilla.com/D115035
By using the soundtouch library, this patch implements the time stretching on the samples in AudioDecoderInputTrack when its playback rate is not 1.0f, in order to support changing playback rate on the captured media stream.
As the time stretcher has to be initialized by a fixed channel count, we would perform a realtime up-mix/down-mix for those audio chunks which have different channel count thant AudioDecoderInputTrack's initial channel count.
Differential Revision: https://phabricator.services.mozilla.com/D114560
When we switch to the DecodedStream from AudioSink. AudioSink would store some frames in its own media queue before those audio frames get played. After AudioSink got shutdown, it doesn't push those frames back to the original audio queue.
Therefore, when DecodedStream starts, it try to retrieve audio data from the audio queue, but it doesn't know that there is already a gap between "the audio DecodedStream is going to play" and "the audio that AudioSink had played".
Differential Revision: https://phabricator.services.mozilla.com/D115749
In the initial implementation, we only used 20 capacity for SPSC queue and added some mechanisms (batched data to save the free space of SPSC queue/clean SPSC queue in each graph iteration) to prevent it from being full.
But that seems not working in some extreme cases where decoding is too fast and the graph thread can't catch up the speef of processing the input. So extend the queue size to 40 that should allow SPSC to work without being full again.
Differential Revision: https://phabricator.services.mozilla.com/D114150
Change the audio track from `SourceMediaTrack` to `AudioDecoderInputTrack`, which allows us to use a pull-based stream track in DecodedStream.
Also a pre-requirement before we start implementing supporting playback rate change on a track captured from the media element.
Differential Revision: https://phabricator.services.mozilla.com/D106042
This patch implements a new track `AudioDecoderInputTrack` which inherits from `ProcessedMediaTrack` and is a pull-based media stream track.
This track would be used for receiving audio decoded data and will support changing playback rate in the future (not yet, will be done in Bug 1517199).
Differential Revision: https://phabricator.services.mozilla.com/D106040
Change the audio track from `SourceMediaTrack` to `AudioDecoderInputTrack`, which allows us to use a pull-based stream track in DecodedStream.
Also a pre-requirement before we start implementing supporting playback rate change on a track captured from the media element.
Differential Revision: https://phabricator.services.mozilla.com/D106042
This patch implements a new track `AudioDecoderInputTrack` which inherits from `ProcessedMediaTrack` and is a pull-based media stream track.
This track would be used for receiving audio decoded data and will support changing playback rate in the future (not yet, will be done in Bug 1517199).
Differential Revision: https://phabricator.services.mozilla.com/D106040
There are no code changes, only #include changes.
It was a fairly mechanical process: Search for all "AUTO_PROFILER_LABEL", and in each file, if only labels are used, convert "GeckoProfiler.h" into "ProfilerLabels.h" (or just add that last one where needed).
In some files, there were also some marker calls but no other profiler-related calls, in these cases "GeckoProfiler.h" was replaced with both "ProfilerLabels.h" and "ProfilerMarkers.h", which still helps in reducing the use of the all-encompassing "GeckoProfiler.h".
Differential Revision: https://phabricator.services.mozilla.com/D104588
When enabling our MediaTrack implementation (which we don't plan to by default,
NB) and disabling all audio tracks and unselecting all video tracks while having
an active captureStream leads to having no output tracks in DecodedStream.
In this case, DecodedStream doesn't know which graph to use for creating the
intermediary tracks it feeds data to. We don't want to resort to the default
graph either, since two graphs on different clocks could then race each other.
With this patch we plumb down a SharedDummyTrack from the media element where
the captureStream was triggered, through MediaDecoder, to DecodedStream. The
SharedDummyTrack guarantees to keep the graph alive, and holds the graph used
for the output tracks.
Differential Revision: https://phabricator.services.mozilla.com/D99822
In the change from bug1674597, AudioSinkWrapper only handles the promise when we succeed opening AudioSink. However, it forgots to handle the promise when fail to start AudioSink.
Differential Revision: https://phabricator.services.mozilla.com/D96738
Audio stream is the class which really know about whether the audio playback is completed or not. So it makes more sense to let it return and manage the promise for playback end, not audio sink.
Differential Revision: https://phabricator.services.mozilla.com/D96468
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
This is essentially the same code as the interleaved version, but backwards,
since the memory layout is the opposite, we want to take advantage of memory
locality, and only touch audio samples once each.
The `ProcessAudioData` method has been renamed, because of the similarity
between the `AudioData` type and `ProcessAudioData`, since it can now process
`AudioBlock`s.
Differential Revision: https://phabricator.services.mozilla.com/D90431
This is essentially the same code as the interleaved version, but backwards,
since the memory layout is the opposite, we want to take advantage of memory
locality, and only touch audio samples once each.
The `ProcessAudioData` method has been renamed, because of the similarity
between the `AudioData` type and `ProcessAudioData`, since it can now process
`AudioBlock`s.
Differential Revision: https://phabricator.services.mozilla.com/D90431
This patch was generated by running:
```
perl -p -i \
-e 's/^(\s+)([a-zA-Z0-9.]+) = NS_ConvertUTF8toUTF16\((.*)\);/\1CopyUTF8toUTF16(\3, \2);/;' \
-e 's/^(\s+)([a-zA-Z0-9.]+) = NS_ConvertUTF16toUTF8\((.*)\);/\1CopyUTF16toUTF8(\3, \2);/;' \
$FILE
```
against every .cpp and .h in mozilla-central, and then fixing up the
inevitable errors that happen as a result of matching C++ expressions with
regexes. The errors fell into three categories:
1. Calling the convert functions with `std::string::c_str()`; these were
changed to simply pass the string instead, relying on implicit conversion
to `mozilla::Span`.
2. Calling the convert functions with raw pointers, which is not permitted
with the copy functions; these were changed to invoke `MakeStringSpan` first.
3. Other miscellaneous errors resulting from over-eager regexes and/or the
replacement not being type-aware. These changes were reverted.
Differential Revision: https://phabricator.services.mozilla.com/D88903
I want to use `PROFILER_MARKER` in the new C++ marker APIs, so I'm renaming this single (recent) use to something else.
Differential Revision: https://phabricator.services.mozilla.com/D87240
This patch will do :
- use `MediaSampleMarkerPayload` to replace `VideoFrameMarkerPayload`
The advantage of doing so :
- after finishing a support for `MediaSampleMarkerPayload` in profiler front-end side, we can show the sample's start and end time when hovering on the marker icon.
Differential Revision: https://phabricator.services.mozilla.com/D75469
This patch will do :
- add `Media` markers in related codes
- implement `MediaSampleMarkerPayload` to record the media sample
The advantage of doing so :
- using markers can help us know what happens on the media by a glance without expanding the call stack
- adding sample markers allows us compare the speed of decoding sample in `MediaDecoderStataMachine` and rendering sample in `VideoSink`
Differential Revision: https://phabricator.services.mozilla.com/D74174
This patch will do :
- create a profiling category `Media`
- add `Media` profiling labels in related codes
The advantage of doing so :
- allow us to easily see what operations are related to media playback from the profiled report
More details :
According to the description in the `ProfilingCategory.h`, `topmost profiler label frame in the label stack determines the category pair of that stack`. Therefore, most labels I added are the first task would run on the thread, in order to ensure all its following tasks can be marked as the media playback label as well.
Differential Revision: https://phabricator.services.mozilla.com/D74171
`AudioSink` provides a way to let us know if the media data is audible, and we should do the same thing for the `DecodedStream`, in order to update the correct audible state.
Differential Revision: https://phabricator.services.mozilla.com/D70259
This patch also tries to remove the event target entirely if it would
default to the main thread on a null event target.
Depends on D67634
Differential Revision: https://phabricator.services.mozilla.com/D67635
--HG--
extra : moz-landing-system : lando
This removes most dependencies on BlocksRingBuffer, to ease the transition to
the upcoming Fission-friendly profile buffer, including:
- Length type,
- SumBytes(),
- Gecko extensions of serialization.
Differential Revision: https://phabricator.services.mozilla.com/D66722
--HG--
rename : tools/profiler/public/BlocksRingBufferGeckoExtensions.h => tools/profiler/public/ProfileBufferEntrySerializationGeckoExtensions.h
extra : moz-landing-system : lando
The new ProfileBufferEntryReader/Writer are now used everywhere, including in
the profilers and tests.
The old EntryReader/Writer have been removed.
Differential Revision: https://phabricator.services.mozilla.com/D65697
--HG--
extra : moz-landing-system : lando
`mPrincipalHandle` and `mPlaying` listeners were being connected in the ctor of a DecodedStream. However, this is not necessary because their attributes will only be modified after the sink start. In addition to that, it was causing problems if a sink was replaced before being started or stopped (and shutdown). This is a valid scenario, though, that we need to support.
Differential Revision: https://phabricator.services.mozilla.com/D63830
--HG--
extra : moz-landing-system : lando
Bug 1600063 changed the chunk-dropping logic from dropping when we had *started*
playing a chunk, to when we had *finished* playing a chunk -- i.e., started
playing the next chunk.
But when a chunk is the very last chunk in the media resource, we cannot start
playing the next chunk, because there is no next chunk. Making this check
inclusive seems reasonable.
Differential Revision: https://phabricator.services.mozilla.com/D57374
--HG--
extra : moz-landing-system : lando
Rather than applying the volume change to the DecodedStream, which won't be
played until the buffer catches up, instead apply the volume to the segments
as they are consumed by the MediaTrackGraph.
Differential Revision: https://phabricator.services.mozilla.com/D56276
--HG--
extra : moz-landing-system : lando
As we added new files, which affects the build bundle created by unified build. Therefore, add missing scope definition and included header to pass the build.
Differential Revision: https://phabricator.services.mozilla.com/D52430
--HG--
extra : moz-landing-system : lando
As we added new files, which affects the build bundle created by unified build. Therefore, add missing scope definition and included header to pass the build.
Differential Revision: https://phabricator.services.mozilla.com/D52430
--HG--
extra : moz-landing-system : lando
Lee Salzman
Paul Adenot
Narcis Beleuzu
Chris Peterson
Gerald Squelart
Cristian Tuns
Andi-Bogdan Postelnicu
Florian Quèze
alwu
Brindusan Cristian
Brendan Early
Simon Giesecke
Bryce Seager van Dyk
Andreas Pehrson
Oleg Skoromnik
Razvan Maries
Mihai Alexandru Michis
Andi-Bogdan Postelnicu
Ricky Stewart
Bogdan Tara
Dorel Luca
Cristina Coroiu
Nathan Froyd
Alex Chronopoulos
Andreas Farre
Jon Bauman
Noemi Erli
Daniel Varga