gecko-dev/content/media/nsBuiltinDecoderStateMachine.h

/* -*- Mode: C++; tab-width: 2; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
/* vim:set ts=2 sw=2 sts=2 et cindent: */
/* ***** BEGIN LICENSE BLOCK *****
 * Version: ML 1.1/GPL 2.0/LGPL 2.1
 *
 * The contents of this file are subject to the Mozilla Public License Version
 * 1.1 (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 * http://www.mozilla.org/MPL/
 *
 * Software distributed under the License is distributed on an "AS IS" basis,
 * WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
 * for the specific language governing rights and limitations under the
 * License.
 *
 * The Original Code is Mozilla code.
 *
 * The Initial Developer of the Original Code is the Mozilla Foundation.
 * Portions created by the Initial Developer are Copyright (C) 2010
 * the Initial Developer. All Rights Reserved.
 *
 * Contributor(s):
 *  Chris Double <chris.double@double.co.nz>
 *  Chris Pearce <chris@pearce.org.nz>
 *
 * Alternatively, the contents of this file may be used under the terms of
 * either the GNU General Public License Version 2 or later (the "GPL"), or
 * the GNU Lesser General Public License Version 2.1 or later (the "LGPL"),
 * in which case the provisions of the GPL or the LGPL are applicable instead
 * of those above. If you wish to allow use of your version of this file only
 * under the terms of either the GPL or the LGPL, and not to allow others to
 * use your version of this file under the terms of the MPL, indicate your
 * decision by deleting the provisions above and replace them with the notice
 * and other provisions required by the GPL or the LGPL. If you do not delete
 * the provisions above, a recipient may use your version of this file under
 * the terms of any one of the MPL, the GPL or the LGPL.
 *
 * ***** END LICENSE BLOCK ***** */
/*
Each video element for a media file has two additional threads beyond
those needed by nsBuiltinDecoder.

  1) The Audio thread writes the decoded audio data to the audio
     hardware.  This is done in a seperate thread to ensure that the
     audio hardware gets a constant stream of data without
     interruption due to decoding or display. At some point
     libsydneyaudio will be refactored to have a callback interface
     where it asks for data and an extra thread will no longer be
     needed.

  2) The decode thread. This thread reads from the media stream and
     decodes the Theora and Vorbis data. It places the decoded data in
     a queue for the other threads to pull from.

All file reads and seeks must occur on either the state machine thread
or the decode thread. Synchronisation is done via a monitor owned by
nsBuiltinDecoder.

The decode thread and the audio thread are created and destroyed in
the state machine thread. When playback needs to occur they are
created and events dispatched to them to start them. These events exit
when decoding is completed or no longer required (during seeking or
shutdown).
    
The decode thread has its own monitor to ensure that its internal
state is independent of the other threads, and to ensure that it's not
hogging the nsBuiltinDecoder monitor while decoding.

a/v synchronisation is handled by the state machine thread. It
examines the audio playback time and compares this to the next frame
in the queue of frames. If it is time to play the video frame it is
then displayed.

Frame skipping is done in the following ways:

  1) The state machine thread will skip all frames in the video queue whose
     display time is less than the current audio time. This ensures
     the correct frame for the current time is always displayed.

  2) The decode thread will stop decoding interframes and read to the
     next keyframe if it determines that decoding the remaining
     interframes will cause playback issues. It detects this by:
       a) If the amount of audio data in the audio queue drops
          below a threshold whereby audio may start to skip.
       b) If the video queue drops below a threshold where it
          will be decoding video data that won't be displayed due
          to the decode thread dropping the frame immediately.

YCbCr conversion is done on the decode thread when it is time to display
the video frame. This means frames that are skipped will not have the
YCbCr conversion done, improving playback.

The decode thread pushes decoded audio and videos frames into two
separate queues - one for audio and one for video. These are kept
separate to make it easy to constantly feed audio data to the sound
hardware while allowing frame skipping of video data. These queues are
threadsafe, and neither the decode, audio, or state machine thread should
be able to monopolize them, and cause starvation of the other threads.

Both queues are bounded by a maximum size. When this size is reached
the decode thread will no longer decode video or audio depending on the
queue that has reached the threshold.

During playback the audio thread will be idle (via a Wait() on the
monitor) if the audio queue is empty. Otherwise it constantly pops an
item off the queue and plays it with a blocking write to the audio
hardware (via nsAudioStream and libsydneyaudio).

The decode thread idles if the video queue is empty or if it is
not yet time to display the next frame.
*/
#if !defined(nsBuiltinDecoderStateMachine_h__)
#define nsBuiltinDecoderStateMachine_h__

#include "prmem.h"
#include "nsThreadUtils.h"
#include "nsBuiltinDecoder.h"
#include "nsBuiltinDecoderReader.h"
#include "nsAudioAvailableEventManager.h"
#include "nsHTMLMediaElement.h"
#include "mozilla/Monitor.h"

/*
  The playback state machine class. This manages the decoding in the
  nsBuiltinDecoderReader on the decode thread, seeking and in-sync-playback on the
  state machine thread, and controls the audio "push" thread.

  All internal state is synchronised via the decoder monitor. NotifyAll
  on the monitor is called when the state of the state machine is changed
  by the main thread. The following changes to state cause a notify:

    mState and data related to that state changed (mSeekTime, etc)
    Metadata Loaded
    First Frame Loaded
    Frame decoded
    data pushed or popped from the video and audio queues

  See nsBuiltinDecoder.h for more details.
*/
class nsBuiltinDecoderStateMachine : public nsDecoderStateMachine
{
public:
  typedef mozilla::Monitor Monitor;
  typedef mozilla::TimeStamp TimeStamp;
  typedef mozilla::TimeDuration TimeDuration;

  nsBuiltinDecoderStateMachine(nsBuiltinDecoder* aDecoder, nsBuiltinDecoderReader* aReader);
  ~nsBuiltinDecoderStateMachine();

  // nsDecoderStateMachine interface
  virtual nsresult Init(nsDecoderStateMachine* aCloneDonor);
  State GetState()
  { 
    mDecoder->GetMonitor().AssertCurrentThreadIn();
    return mState; 
  }
  virtual void SetVolume(float aVolume);
  virtual void Shutdown();
  virtual PRInt64 GetDuration();
  virtual void SetDuration(PRInt64 aDuration);
  virtual PRBool OnDecodeThread() {
    return IsCurrentThread(mDecodeThread);
  }

  virtual nsHTMLMediaElement::NextFrameStatus GetNextFrameStatus();
  virtual void Decode();
  virtual void Seek(float aTime);
  virtual float GetCurrentTime();
  virtual void ClearPositionChangeFlag();
  virtual void SetSeekable(PRBool aSeekable);
  virtual void UpdatePlaybackPosition(PRInt64 aTime);
  virtual void StartBuffering();


  // Load metadata Called on the state machine thread. The decoder monitor must be held with
  // exactly one lock count.
  virtual void LoadMetadata();

  // State machine thread run function. Polls the state, sends frames to be
  // displayed at appropriate times, and generally manages the decode.
  NS_IMETHOD Run();

  // This is called on the state machine thread and audio thread.
  // The decoder monitor must be obtained before calling this.
  PRBool HasAudio() const {
    mDecoder->GetMonitor().AssertCurrentThreadIn();
    return mReader->GetInfo().mHasAudio;
  }

  // This is called on the state machine thread and audio thread.
  // The decoder monitor must be obtained before calling this.
  PRBool HasVideo() const {
    mDecoder->GetMonitor().AssertCurrentThreadIn();
    return mReader->GetInfo().mHasVideo;
  }

  // Should be called by main thread.
  PRBool HaveNextFrameData() const;

  // Must be called with the decode monitor held.
  PRBool IsBuffering() const {
    mDecoder->GetMonitor().AssertCurrentThreadIn();

    return mState == nsBuiltinDecoderStateMachine::DECODER_STATE_BUFFERING;
  }

  // Must be called with the decode monitor held.
  PRBool IsSeeking() const {
    mDecoder->GetMonitor().AssertCurrentThreadIn();

    return mState == nsBuiltinDecoderStateMachine::DECODER_STATE_SEEKING;
  }

  // Functions used by assertions to ensure we're calling things
  // on the appropriate threads.
  PRBool OnAudioThread() {
    return IsCurrentThread(mAudioThread);
  }

  PRBool OnStateMachineThread() {
    return mDecoder->OnStateMachineThread();
  }

  // Decode loop, called on the decode thread.
  void DecodeLoop();

  // The decoder object that created this state machine. The decoder
  // always outlives us since it controls our lifetime. This is accessed
  // read only on the AV, state machine, audio and main thread.
  nsBuiltinDecoder* mDecoder;

  // The decoder monitor must be obtained before modifying this state.
  // NotifyAll on the monitor must be called when the state is changed by
  // the main thread so the decoder thread can wake up.
  // Accessed on state machine, audio, main, and AV thread. 
  State mState;

  nsresult GetBuffered(nsTimeRanges* aBuffered) {
    NS_ASSERTION(NS_IsMainThread(), "Only call on main thread");
    return mReader->GetBuffered(aBuffered, mStartTime);
  }

  void NotifyDataArrived(const char* aBuffer, PRUint32 aLength, PRUint32 aOffset) {
    NS_ASSERTION(NS_IsMainThread(), "Only call on main thread");
    mReader->NotifyDataArrived(aBuffer, aLength, aOffset);
  }

  PRInt64 GetEndMediaTime() const {
    mDecoder->GetMonitor().AssertCurrentThreadIn();
    return mEndTime;
  }

protected:

  // Returns PR_TRUE if the decode is withing an estimated one tenth of a
  // second's worth of data of the download, i.e. the decode has almost
  // caught up with the download. If we can't estimate one tenth of a second's
  // worth of data, we'll return PR_TRUE if the decode is within 100KB of
  // the download.
  PRBool IsDecodeCloseToDownload();

  // Returns the number of unplayed ms of audio we've got decoded and/or
  // pushed to the hardware waiting to play. This is how much audio we can
  // play without having to run the audio decoder.
  PRInt64 AudioDecodedMs() const;

  // Returns PR_TRUE when there's decoded audio waiting to play.
  // The decoder monitor must be held.
  PRBool HasFutureAudio() const;

  // Waits on the decoder Monitor for aMs. If the decoder monitor is awoken
  // by a Notify() call, we'll continue waiting, unless we've moved into
  // shutdown state. This enables us to ensure that we wait for a specified
  // time, and that the myriad of Notify()s we do an the decoder monitor
  // don't cause the audio thread to be starved. The decoder monitor must
  // be locked.
  void Wait(PRUint32 aMs);

  // Dispatches an asynchronous event to update the media element's ready state.
  void UpdateReadyState();

  // Resets playback timing data. Called when we seek, on the state machine
  // thread.
  void ResetPlayback();

  // Returns the audio clock, if we have audio, or -1 if we don't.
  // Called on the state machine thread.
  PRInt64 GetAudioClock();

  // Returns the presentation time of the first sample or frame in the media.
  // If the media has video, it returns the first video frame. The decoder
  // monitor must be held with exactly one lock count. Called on the state
  // machine thread.
  VideoData* FindStartTime();

  // Finds the end time of the last frame of data in the file, storing the value
  // in mEndTime if successful. The decoder must be held with exactly one lock
  // count. Called on the state machine thread.
  void FindEndTime();

  // Performs YCbCr to RGB conversion, and pushes the image down the
  // rendering pipeline. Called on the state machine thread.
  void RenderVideoFrame(VideoData* aData);

  // If we have video, display a video frame if it's time for display has
  // arrived, otherwise sleep until it's time for the next sample. Update
  // the current frame time as appropriate, and trigger ready state update.
  // The decoder monitor must be held with exactly one lock count. Called
  // on the state machine thread.
  void AdvanceFrame();

  // Pushes up to aSamples samples of silence onto the audio hardware. Returns
  // the number of samples acutally pushed to the hardware. This pushes up to
  // 32KB worth of samples to the hardware before returning, so must be called
  // in a loop to ensure that the desired number of samples are pushed to the
  // hardware. This ensures that the playback position advances smoothly, and
  // guarantees that we don't try to allocate an impossibly large chunk of
  // memory in order to play back silence. Called on the audio thread.
  PRUint32 PlaySilence(PRUint32 aSamples, PRUint32 aChannels,
                       PRUint64 aSampleOffset);

  // Pops an audio chunk from the front of the audio queue, and pushes its
  // sound data to the audio hardware. MozAudioAvailable sample data is also
  // queued here. Called on the audio thread.
  PRUint32 PlayFromAudioQueue(PRUint64 aSampleOffset, PRUint32 aChannels);

  // Stops the decode threads. The decoder monitor must be held with exactly
  // one lock count. Called on the state machine thread.
  void StopDecodeThreads();

  // Starts the decode threads. The decoder monitor must be held with exactly
  // one lock count. Called on the state machine thread.
  nsresult StartDecodeThreads();

  // The main loop for the audio thread. Sent to the thread as
  // an nsRunnableMethod. This continually does blocking writes to
  // to audio stream to play audio data.
  void AudioLoop();

  // Stop or pause playback of media. This has two modes, denoted by
  // aMode being either AUDIO_PAUSE or AUDIO_SHUTDOWN.
  //
  // AUDIO_PAUSE: Suspends the audio stream to be resumed later.
  // This does not close the OS based audio stream 
  //
  // AUDIO_SHUTDOWN: Closes and destroys the audio stream and
  // releases any OS resources.
  //
  // The decoder monitor must be held with exactly one lock count. Called
  // on the state machine thread.
  enum eStopMode {AUDIO_PAUSE, AUDIO_SHUTDOWN};
  void StopPlayback(eStopMode aMode);

  // Resume playback of media. Must be called with the decode monitor held.
  // This resumes a paused audio stream. The decoder monitor must be held with
  // exactly one lock count. Called on the state machine thread.
  void StartPlayback();

  // Returns PR_TRUE if we're currently playing. The decoder monitor must
  // be held.
  PRBool IsPlaying();

  // Returns the "media time". This is the absolute time which the media
  // playback has reached. i.e. this returns values in the range
  // [mStartTime, mEndTime], and mStartTime will not be 0 if the media does
  // not start at 0. Note this is different to the value returned
  // by GetCurrentTime(), which is in the range [0,duration].
  PRInt64 GetMediaTime() const {
    mDecoder->GetMonitor().AssertCurrentThreadIn();
    return mStartTime + mCurrentFrameTime;
  }

  // Monitor on mAudioStream. This monitor must be held in order to delete
  // or use the audio stream. This stops us destroying the audio stream
  // while it's being used on another thread (typically when it's being
  // written to on the audio thread).
  Monitor mAudioMonitor;

  // The size of the decoded YCbCr frame.
  // Accessed on state machine thread.
  PRUint32 mCbCrSize;

  // Accessed on state machine thread.
  nsAutoArrayPtr<unsigned char> mCbCrBuffer;

  // Thread for pushing audio onto the audio hardware.
  // The "audio push thread".
  nsCOMPtr<nsIThread> mAudioThread;

  // Thread for decoding video in background. The "decode thread".
  nsCOMPtr<nsIThread> mDecodeThread;

  // The time that playback started from the system clock. This is used
  // for timing the display of audio frames when there's no audio.
  // Accessed only via the state machine thread.
  TimeStamp mPlayStartTime;

  // The amount of time we've spent playing already the media. The current
  // playback position is therefore (mPlayDuration + (now - mPlayStartTime)).
  // Accessed only via the state machine thread.
  TimeDuration mPlayDuration;

  // Time that buffering started. Used for buffering timeout and only
  // accessed on the state machine thread.
  TimeStamp mBufferingStart;

  // Download position where we should stop buffering. Only
  // accessed on the state machine thread.
  PRInt64 mBufferingEndOffset;

  // Start time of the media, in milliseconds. This is the presentation
  // time of the first sample decoded from the media, and is used to calculate
  // duration and as a bounds for seeking. Accessed on state machine and
  // main thread. Access controlled by decoder monitor.
  PRInt64 mStartTime;

  // Time of the last page in the media, in milliseconds. This is the
  // end time of the last sample in the media. Accessed on state
  // machine and main thread. Access controlled by decoder monitor.
  PRInt64 mEndTime;

  // Position to seek to in milliseconds when the seek state transition occurs.
  // The decoder monitor lock must be obtained before reading or writing
  // this value. Accessed on main and state machine thread.
  PRInt64 mSeekTime;

  // The audio stream resource. Used on the state machine, audio, and main
  // threads. You must hold the mAudioMonitor, and must NOT hold the decoder
  // monitor when using the audio stream!
  nsRefPtr<nsAudioStream> mAudioStream;

  // The reader, don't call its methods with the decoder monitor held.
  // This is created in the play state machine's constructor, and destroyed
  // in the play state machine's destructor.
  nsAutoPtr<nsBuiltinDecoderReader> mReader;

  // The time of the current frame in milliseconds. This is referenced from
  // 0 which is the initial playback position. Set by the state machine
  // thread, and read-only from the main thread to get the current
  // time value. Synchronised via decoder monitor.
  PRInt64 mCurrentFrameTime;

  // The presentation time of the first audio sample that was played. We can
  // add this to the audio stream position to determine the current audio time.
  // Accessed on audio and state machine thread. Synchronized by decoder monitor.
  PRInt64 mAudioStartTime;

  // The end time of the last audio sample that's been pushed onto the audio
  // hardware. This will approximately be the end time of the audio stream,
  // unless another sample is pushed to the hardware.
  PRInt64 mAudioEndTime;

  // The presentation end time of the last video frame which has been displayed.
  // Accessed from the state machine thread.
  PRInt64 mVideoFrameEndTime;
  
  // Volume of playback. 0.0 = muted. 1.0 = full volume. Read/Written
  // from the state machine and main threads. Synchronised via decoder
  // monitor.
  float mVolume;

  // PR_TRUE if the media resource can be seeked. Accessed from the state
  // machine and main threads. Synchronised via decoder monitor.
  PRPackedBool mSeekable;

  // PR_TRUE if an event to notify about a change in the playback
  // position has been queued, but not yet run. It is set to PR_FALSE when
  // the event is run. This allows coalescing of these events as they can be
  // produced many times per second. Synchronised via decoder monitor.
  // Accessed on main and state machine threads.
  PRPackedBool mPositionChangeQueued;

  // PR_TRUE if the audio playback thread has finished. It is finished
  // when either all the audio samples in the Vorbis bitstream have completed
  // playing, or we've moved into shutdown state, and the threads are to be
  // destroyed. Written by the audio playback thread and read and written by
  // the state machine thread. Synchronised via decoder monitor.
  PRPackedBool mAudioCompleted;

  // PR_TRUE if mDuration has a value obtained from an HTTP header, or from
  // the media index/metadata. Accessed on the state machine thread.
  PRPackedBool mGotDurationFromMetaData;
    
  // PR_FALSE while decode threads should be running. Accessed on audio, 
  // state machine and decode threads. Syncrhonised by decoder monitor.
  PRPackedBool mStopDecodeThreads;

private:
  // Manager for queuing and dispatching MozAudioAvailable events.  The
  // event manager is accessed from the state machine and audio threads,
  // and takes care of synchronizing access to its internal queue.
  nsAudioAvailableEventManager mEventManager;
};

#endif