Stats overlay implementation.

src/api/04-enhancements/11-playback-performance.md

@@ -0,0 +1,100 @@
+---
+title: Playback Performance
+description: |
+  Use the Media Capabilities API and the Video Playback Quality API to estimate and measure playback efficiency and smoothness.
+date: February 18, 2022
+length: '1:04'
+video-sources:
+  - src: https://storage.googleapis.com/kino-assets/google-cast/manifest.mpd
+    type: application/dash+xml
+    cast: true
+  - src: https://storage.googleapis.com/kino-assets/google-cast/master.m3u8
+    type: application/x-mpegURL
+video-subtitles:
+  - default: true
+    kind: captions
+    label: English
+    src: https://storage.googleapis.com/kino-assets/google-cast/cap-en.vtt
+    srclang: en
+  - default: false
+    kind: captions
+    label: Česky
+    src: https://storage.googleapis.com/kino-assets/google-cast/cap-cs.vtt
+    srclang: cz
+thumbnail: https://storage.googleapis.com/kino-assets/google-cast/thumbnail.png
+media-session-artwork:
+  - sizes: 96x96
+    src: https://storage.googleapis.com/kino-assets/google-cast/artwork-96x96.png
+    type: image/png
+  - sizes: 128x128
+    src: https://storage.googleapis.com/kino-assets/google-cast/artwork-128x128.png
+    type: image/png
+  - sizes: 192x192
+    src: https://storage.googleapis.com/kino-assets/google-cast/artwork-192x192.png
+    type: image/png
+  - sizes: 256x256
+    src: https://storage.googleapis.com/kino-assets/google-cast/artwork-256x256.png
+    type: image/png
+  - sizes: 384x384
+    src: https://storage.googleapis.com/kino-assets/google-cast/artwork-384x384.png
+    type: image/png
+  - sizes: 512x512
+    src: https://storage.googleapis.com/kino-assets/google-cast/artwork-512x512.png
+    type: image/png
+stats: true
+---
+
+## Introduction
+
+The simplest way of embedding a video on the web is using the HTML Video
+element (`<video>`). However, every browser supports a different set of video
+formats. Picking the right format will ensure your video has the highest
+possible browser compatibility and is going to play in almost every situation.
+
+## Choosing the right format
+
+Browsers usually support multiple video formats, but there is a limited overlap
+in their capabilities. If you only want to encode your videos in a single
+format, the safest choice is to use an [MP4 container] encapsulating `H.264`
+video and `AAC` audio.
+
+### Basic code example
+
+```html
+<video controls>
+  <source src="video.mp4" type="video/mp4">
+</video>
+```
+
+This simple approach is enough to get you up and running. Your video will now
+play in all major web browsers. Notice the `controls` attribute that instructs
+browsers to allow users to control video playback, which includes volume,
+seeking, selecting captions, pause/resume playback etc.
+
+A single video source is simple to maintain, but it gives rise to some
+challenges. Users of your site are going to use different classes of devices
+to watch the video. A high resolution video is going to look great on desktop,
+but likely will take a long time to load on slower cellular networks.
+
+Take the video at the top of this page as an example, which has a `1280×720`
+resolution using the `H.264` codec with an effective bit rate of `1503 kb/s`.
+It looks decent on desktop while being small enough to not cause stuttering on
+good quality 3G networks. The video is clear, however, it's not a perfect fit
+for either use case and is why you should probably provide multiple video
+sources with a bitrate targeting the needs of your users.
+
+### Example FFmpeg command
+
+```
+ffmpeg -i source.mp4 -b:v 1350k -c:v libx264 -c:a copy -filter:v "scale=1280:-1" -preset veryslow video.mp4
+```
+
+## What's Next?
+
+Advanced codecs like `VP9` and `HEVC` generally produce smaller file sizes,
+which improves the visual quality and/or experience on slower networks. Next,
+we'll learn when and how to specify [multiple sources] within the HTML5
+`<video>` element.
+
+[MP4 container]: https://caniuse.com/mpeg4
+[multiple sources]: /multiple-sources/

src/api/04-enhancements/index.md

@@ -1,6 +1,6 @@
 ---
-name: Technical APIs
-slug: technical-apis
+name: Enhancements
+slug: enhancements
 ---
 
 Make your video applications even more robust by making use of several more advanced media APIs implemented in the browsers today.

src/js/constants.js

@@ -137,3 +137,9 @@ export const PIP_CLASSNAME = 'picture-in-picture';
 export const CAST_CLASSNAME = 'cast';
 export const CAST_HAS_TARGET_NAME = 'cast-has-target';
 export const CAST_TARGET_NAME = 'cast-target-name';
+
+/**
+ * Stats overlay.
+ */
+export const STATS_OVERLAY_CLASSNAME = 'stats-overlay';
+export const STATS_OVERLAY_DISPLAYED_CLASSNAME = 'stats-overlay-visible';

src/js/sw/cache.js

@@ -47,4 +47,5 @@ export default [
   'https://storage.googleapis.com/kino-assets/picture-in-picture/thumbnail.png',
   'https://storage.googleapis.com/kino-assets/google-cast/thumbnail.png',
   'https://storage.googleapis.com/kino-assets/google-cast/thumbnail.png',
+  'https://storage.googleapis.com/kino-assets/google-cast/thumbnail.png',
 ];

src/js/web-components/video-player/VideoPlayer.css

@@ -82,6 +82,36 @@ video {
   display: inline;
 }
 
+.stats-overlay {
+  display: none;
+  position: absolute;
+  inset: 0;
+  right: auto;
+  bottom: auto;
+  max-height: 65%;
+  max-width: 100%;
+  overflow: auto;
+  z-index: 1;
+  background-color: var(--code-background);
+  color: var(--icon);
+  padding: 1em;
+  opacity: 0.9;
+  box-sizing: border-box;
+}
+
+.stats-overlay h4 {
+  margin-top: 1em;
+  margin-bottom: 0.5em;
+}
+
+.stats-overlay h4:first-child {
+  margin-top: 0;
+}
+
+:host(.stats-overlay-visible) .stats-overlay {
+  display: block;
+}
+
 @media (min-width: 720px) {
   :host(.picture-in-picture) .pip-overlay {
     row-gap: 32px;
@@ -89,4 +119,9 @@ video {
   :host(.cast) .cast-overlay {
     row-gap: 32px;
   }
+  .stats-overlay {
+    inset: 1em;
+    right: auto;
+    bottom: auto;
+  }
 }

src/js/web-components/video-player/VideoPlayer.js

@@ -25,8 +25,12 @@ import {
   CAST_TARGET_NAME,
   MEDIA_SESSION_DEFAULT_ARTWORK,
   PIP_CLASSNAME,
+  STATS_OVERLAY_CLASSNAME,
+  STATS_OVERLAY_DISPLAYED_CLASSNAME,
 } from '../../constants';
 import decryptVideo from '../../utils/decryptVideo';
+import { getMediaConfigurationVideo } from '../../utils/getMediaConfiguration';
+import getDecodingInfo from '../../utils/getDecodingInfo';
 
 export default class extends HTMLElement {
   /**
@@ -119,6 +123,7 @@ export default class extends HTMLElement {
       <svg viewBox="0 0 128 128" xmlns="http://www.w3.org/2000/svg" xml:space="preserve" fill-rule="evenodd" clip-rule="evenodd" stroke-linecap="round" stroke-miterlimit="10"><path d="M34.133 107.201c0-13.253-10.747-24-24-24M53.333 107.2c0-23.861-19.339-43.2-43.2-43.2" fill="none" stroke="var(--icon)" stroke-width="8"/><path d="M10.133 112.001a4.8 4.8 0 1 0 0-9.6 4.8 4.8 0 0 0 0 9.6Z" fill="var(--icon)" fill-rule="nonzero"/><path d="M5.333 49.778V32c0-5.891 4.776-10.667 10.667-10.667h96c5.891 0 10.667 4.776 10.667 10.667v64c0 5.891-4.776 10.667-10.667 10.667H72.381" fill="none" stroke="var(--icon)" stroke-width="8"/></svg>
       <p>Casting<span class="cast-target"> to <span class="cast-target-name"></span></span></p>
     </div>
+    <div class="stats-overlay">STATS OVERLAY</div>
     `;
 
     while (this.internal.root.firstChild) {
@@ -161,6 +166,20 @@ export default class extends HTMLElement {
     this.videoElement.addEventListener('play', () => {
       if (!this.internal.mediaSessionIsInit) this.initMediaSession();
     });
+
+    /**
+     * Set up the stats overlay.
+     */
+    if (this.internal.videoData.stats) {
+      this.videoElement.addEventListener(
+        'play',
+        () => this.classList.add(STATS_OVERLAY_DISPLAYED_CLASSNAME),
+        { once: true },
+      );
+
+      this.videoElement.addEventListener('playing', this.updateStatsOverlay.bind(this));
+      this.videoElement.addEventListener('timeupdate', this.updateStatsOverlay.bind(this));
+    }
   }
 
   /**
@@ -510,4 +529,57 @@ export default class extends HTMLElement {
       this.classList.remove(CAST_CLASSNAME);
     }
   }
+
+  /**
+   * Updates the video playback statistics rendered on top of the video.
+   */
+  async updateStatsOverlay() {
+    let capText = '';
+    let vqText = '';
+    let mediaText = '';
+
+    const statsOverlayEl = this.internal.root.querySelector(`.${STATS_OVERLAY_CLASSNAME}`);
+
+    /** @type {VideoPlaybackQuality} */
+    const vq = this.videoElement.getVideoPlaybackQuality();
+    const vqData = [
+      ['Total frames: ', vq.totalVideoFrames],
+      ['Dropped frames: ', vq.droppedVideoFrames],
+    ];
+    vqText = vqData.map(([label, value]) => `<div>${label}${value}</div>`).join('');
+
+    const reps = this.internal.streamer?.stream?.media?.representations;
+    const selectedReps = this.internal.streamer?.stream?.media?.lastRepresentationsIds;
+
+    if (reps && selectedReps) {
+      const videoId = selectedReps.video;
+      const videoRep = reps.video.find((rep) => rep.id === videoId);
+
+      if (videoRep) {
+        const videoConfiguration = getMediaConfigurationVideo(videoRep);
+        const decodingInfo = await getDecodingInfo(videoConfiguration);
+
+        const capData = [
+          ['Power efficient: ', decodingInfo.powerEfficient],
+          ['Smooth: ', decodingInfo.smooth],
+          ['Supported: ', decodingInfo.supported],
+        ];
+        capText = capData.map(([label, value]) => `<div>${label}${value}</div>`).join('');
+
+        const mediaData = [
+          ['Video codec: ', videoConfiguration.video.contentType],
+          ['Video resolution: ', `${videoConfiguration.video.width}x${videoConfiguration.video.height}`],
+        ];
+        mediaText = mediaData.map(([label, value]) => `<div>${label}${value}</div>`).join('');
+      }
+    }
+
+    statsOverlayEl.innerHTML = `
+      <h4>Media Info</h4>
+      ${mediaText}
+      <h4>Video Playback Quality API</h4>
+      ${vqText}
+      <h4>Media Capabilities API</h4>
+      ${capText}`;
+  }
 }