Bug 1722171 - use time threshold to determine if we need to skip to next key frame. r=bryce

In bug1692881, we introduced a mechanism to skip to next key frame if video decoding is too slow, which aims to help slow decoding issue on high resolution videos. Eg. 4k+.

However, there are some cases which would also lead to slow decoding. Eg. when internal seek (seek on one track only) happens on a place where we can't seek to directly, we need to seek to the previous keyframe and then decode all of them until we get the target frame.

That could lead to spending huge amount of time if the target frame is far from the keyframe, and causes decoded video frame later than the media time because audio won't stop playing during video seek. However, in above example, there would only be few frames later than the media time, and the video will soon catch up with the time.

As we only want to trigger this mechanism for continually slow decoding problem, and completely removing this mechanism would lead us fallback to the situation where some 4K+ videos would be frozen all the time during the playback.

In this patch, we introduce a method which still keep this mechanism but won't perform that as aggressive as before, this uses the time threshold to seek to the next keyframe when video has been late for too long.

Differential Revision: https://phabricator.services.mozilla.com/D126858

dom/media/MediaDecoderStateMachine.cpp

@@ -593,11 +593,13 @@ class MediaDecoderStateMachine::DecodingState
   void HandleVideoDecoded(VideoData* aVideo) override {
     const auto currentTime = mMaster->GetMediaTime();
     if (aVideo->GetEndTime() < currentTime) {
-      SLOG("video %" PRId64 " is too late (current=%" PRId64 ")",
-           aVideo->GetEndTime().ToMicroseconds(), currentTime.ToMicroseconds());
-      mRequestNextVideoKeyFrame = true;
+      if (!mVideoFirstLateTime) {
+        mVideoFirstLateTime = Some(TimeStamp::Now());
+      }
+      SLOG("video %" PRId64 " starts being late (current=%" PRId64 ")",
+           aVideo->mTime.ToMicroseconds(), currentTime.ToMicroseconds());
     } else {
-      mRequestNextVideoKeyFrame = false;
+      mVideoFirstLateTime.reset();
     }
     mMaster->PushVideo(aVideo);
     DispatchDecodeTasksIfNeeded();
@@ -608,7 +610,7 @@ class MediaDecoderStateMachine::DecodingState
 
   void HandleVideoCanceled() override {
     mMaster->RequestVideoData(mMaster->GetMediaTime(),
-                              mRequestNextVideoKeyFrame);
+                              ShouldRequestNextKeyFrame());
   }
 
   void HandleEndOfAudio() override;
@@ -630,7 +632,7 @@ class MediaDecoderStateMachine::DecodingState
 
   void HandleVideoWaited(MediaData::Type aType) override {
     mMaster->RequestVideoData(mMaster->GetMediaTime(),
-                              mRequestNextVideoKeyFrame);
+                              ShouldRequestNextKeyFrame());
   }
 
   void HandleAudioCaptured() override {
@@ -660,6 +662,7 @@ class MediaDecoderStateMachine::DecodingState
 
     if (aPlayState == MediaDecoder::PLAY_STATE_PAUSED) {
       StartDormantTimer();
+      mVideoFirstLateTime.reset();
     } else {
       mDormantTimer.Reset();
     }
@@ -758,6 +761,23 @@ class MediaDecoderStateMachine::DecodingState
         [this]() { mDormantTimer.CompleteRequest(); });
   }
 
+  bool ShouldRequestNextKeyFrame() const {
+    if (!mVideoFirstLateTime) {
+      return false;
+    }
+    const double elapsedTimeMs =
+        (TimeStamp::Now() - *mVideoFirstLateTime).ToMilliseconds();
+    const bool rv = elapsedTimeMs >=
+                    StaticPrefs::media_decoder_skip_when_video_too_slow_ms();
+    if (rv) {
+      SLOG(
+          "video has been late behind media time for %f ms, should skip to "
+          "next key frame",
+          elapsedTimeMs);
+    }
+    return rv;
+  }
+
   // Time at which we started decoding.
   TimeStamp mDecodeStartTime;
 
@@ -776,10 +796,10 @@ class MediaDecoderStateMachine::DecodingState
   MediaEventListener mOnAudioPopped;
   MediaEventListener mOnVideoPopped;
 
-  // It will be set to true if the received decoded video frame is already late
-  // comparing to the current time. So we will want to directly request the next
-  // keyframe in order to catch up with the playback time.
-  bool mRequestNextVideoKeyFrame = false;
+  // If video has been later than the media time, this will records when the
+  // video started being late. It will be reset once video catches up with the
+  // media time.
+  Maybe<TimeStamp> mVideoFirstLateTime;
 };
 
 /**
@@ -2477,7 +2497,8 @@ void MediaDecoderStateMachine::DecodingState::EnsureVideoDecodeTaskQueued() {
       mMaster->IsWaitingVideoData()) {
     return;
   }
-  mMaster->RequestVideoData(mMaster->GetMediaTime(), mRequestNextVideoKeyFrame);
+  mMaster->RequestVideoData(mMaster->GetMediaTime(),
+                            ShouldRequestNextKeyFrame());
 }
 
 void MediaDecoderStateMachine::DecodingState::MaybeStartBuffering() {

modules/libpref/init/StaticPrefList.yaml

@@ -8313,6 +8313,20 @@
   value: true
   mirror: always
 
+# When video continues later than the current media time for this period of
+# time, then it will trigger skip-to-next-keyframe mechanism. As this aims to
+# solve the drop frames issue where video decoding too slow for high
+# resolution videos. eg. 4k+. Therefore, the value is is determined by the
+# telemetry probe `video_inter_keyframe_max_ms` in the key of `AV,h>2160` which
+# shows that 95% video's key frame interval are less than ~5000. We use its
+# half value as a threashold to decide whether we should keep decoding in the
+# current video position or jump to the next keyframe in order to decode future
+# frames in advance.
+- name: media.decoder.skip_when_video_too_slow_ms
+  type: RelaxedAtomicInt32
+  value: 2500
+  mirror: always
+
 - name: media.gmp.decoder.enabled
   type: RelaxedAtomicBool
   value: false