Merge branch 'master' into stable

This commit is contained in:
Makoto Inoue 2008-10-20 16:38:20 +01:00
Родитель 24e07910a6 58090cbb4d
Коммит 55e206ae67
19 изменённых файлов: 247 добавлений и 344 удалений

140
README
Просмотреть файл

@ -1,140 +0,0 @@
Panda is a Merb app which runs on a special EC2 instance to encode videos for you. Uploaded videos are stored on S3 with a small amount of info kept in SimpleDB. The REST API makes it easy to integrate user video uploading into your web application.
How does Panda work?
====================
1. Video is uploaded to panda
2. Panda checks the video's metadata, uploads the raw file to S3 and adds it to the encoding queue
3. The encoder application picks the encoding job off the queue when it's free and encodes the video to all possible formats
4. Panda sends a callback to your web application notifying you the video has been encoded
5. You use the S3 url of the encoding you want to show to your users
Installation and setup
======================
Gems
----
Merb 0.9.2
S3 (http://amazon.rubyforge.org/)
SimpleDB (http://nytimes.rubyforge.org/amazon_sdb/)
gem install merb merb_helpers activesupport RubyInline amazon_sdb SQS uuid
Other deps
----------
# libjpeg
http://www.ijg.org/
http://www.libgd.org/FAQ#gd_keeps_saying_it_can.27t_find_png_or_jpeg_support._I_did_install_libpng_and_libjpeg._What_am_I_missing.3F
Grab the source, build, and install
mkdir -p ~/src && cd ~/src
wget ftp://ftp.uu.net/graphics/jpeg/jpegsrc.v6b.tar.gz
tar zxvf jpegsrc.v6b.tar.gz
cd jpeg-6b && ./configure && make && sudo make install
# libgd
http://www.libgd.org/Main_Page
Grab the source, build, and install
mkdir -p ~/src && cd ~/src
wget http://www.libgd.org/releases/gd-2.0.35.tar.gz
tar zxvf gd-2.0.35.tar.gz
cd gd-2.0.35 && ./configure && make && sudo make install
RVideo
------
Grab the source and build the gem:
git clone git://github.com/jaikoo/rvideo.git
cd rvideo
rake install_gem
Install the rvideo tools (on OS X at least; your system may differ)
sudo cp lib/rvideo/tools/*.rb /usr/lib/ruby/gems/1.8/gems/rvideo-0.9.4/lib/rvideo/tools/
FFMPEG
-----
Available in all good package repositories including Darwin Ports.
If you're developing on Mac OS X, you can grab ffmpeg out of the ffmpegX application instead of compiling it: http://www.macosxhints.com/article.php?story=20061220082125312
Configuration
=============
Encoding profiles
-----------------
Profile.create!(:title => "MP4 SD", :container => "mp4", :video_codec => "mpeg4", :video_bitrate => 300, :audio_codec => "aac", :audio_bitrate => 48, :audio_sample_rate => 48000, :width => 320, :height => 240, :fps => 24, :position => 2, :player => nil)
Profile.create!(:title => "Flash video SD", :container => "flv", :video_bitrate => 300, :audio_bitrate => 48, :width => 320, :height => 240, :fps => 24, :position => 0, :player => "flash")
Profile.create!(:title => "Flash video HI", :container => "flv", :video_bitrate => 400, :audio_bitrate => 48, :width => 480, :height => 360, :fps => 24, :position => 1, :player => "flash")
Profile.create!(:title => "Flash h264 SD", :container => "mp4", :video_bitrate => 300, :audio_codec => "aac", :audio_bitrate => 48, :width => 320, :height => 240, :fps => 24, :position => 2, :player => "flash")
Profile.create!(:title => "Flash h264 HI", :container => "mp4", :video_bitrate => 400, :audio_bitrate => 48, :audio_coded => "aac", :width => 480, :height => 360, :fps => 24, :position => 3, :player => "flash")
Profile.create!(:title => "Flash h264 480p", :container => "mp4", :video_bitrate => 600, :audio_bitrate => 48, :audio_coded => "aac", :width => 852, :height => 480, :fps => 24, :position => 4, :player => "flash")
Further information
===================
Investigating encoding errors
-----------------------------
When an encoding fails the status of the video is set to 'error' and the output of ffmpeg is saved on S3 with the filename video_token.error
SimpleDB schemas
================
Videos
------
filename # 976a4b00-16cc-012b-7316-001ec2b5c0e1.flv
original_filename # sneezing_panda.flv
parent
status # original, queued, processing, done, error
duration
container
width
height
video_codec
video_bitrate
fps
audio_codec
audio_sample_rate
profile # id of encoding profile used
profile_title
updated_at
created_at
Encoding profiles
-----------------
title
container
width
height
video_codec
video_bitrate
fps
audio_codec
audio_bitrate
audio_sample_rate
updated_at
created_at

169
README.markdown Normal file
Просмотреть файл

@ -0,0 +1,169 @@
Panda
=====
Panda is an open source solution for video uploading, encoding and streaming.
Please see [pandastream.com](http://pandastream.com/) for an introduction and lots of documentation.
Information beyond this point is aimed at people who want to contribute to panda and / or understand how it works.
How does Panda work?
====================
1. Video is uploaded to panda
2. Panda checks the video's metadata, uploads the raw file to S3 and adds it to the encoding queue
3. The encoder application picks the encoding job off the queue when it's free and encodes the video to all possible formats
4. Panda sends a callback to your web application notifying you the video has been encoded
5. You use the appropriate S3 url of the encoding to embed the video
Installation and setup
======================
Please note that this guide has only been tested on OSX. Please post modifications to the google group if you try another platform.
Install dependencies
--------------------
First install the following gems
sudo gem install merb merb_helpers activesupport RubyInline amazon_sdb aws-s3 uuid flvtool2
Next you'll need to install libgd, ffmpeg, and the rvideo gem:
### [LibGD](http://www.libgd.org/Main_Page)
#### On OSX (using macports)
Installing gd2 with macports seems to be the fastest way to install the required dependencies:
sudo port install gd2
For some reason (which I've not really looked into) the macports install doesn't quite install everything so you still need to install libgd itself (but not the dependencies) from source:
mkdir -p ~/src && cd ~/src
curl http://www.libgd.org/releases/gd-2.0.35.tar.gz > gd-2.0.35.tar.gz
tar zxvf gd-2.0.35.tar.gz
cd gd-2.0.35 && ./configure && make && sudo make install
Alternatively, if you want to build everything from source, see [this tutorial](http://mikewest.org/archive/installing-libgd-from-source-on-os-x). Here's a [possible gotcha](http://www.libgd.org/FAQ#gd_keeps_saying_it_can.27t_find_png_or_jpeg_support._I_did_install_libpng_and_libjpeg._What_am_I_missing.3F).
#### Other platforms
The following libraries are required by libgd:
* [zlib](http://www.zlib.net)
* [libjpeg](http://www.ijg.org/)
* [libgd](http://www.libgd.org/Main_Page)
Install [libjpeg](http://www.ijg.org/) from source:
mkdir -p ~/src && cd ~/src
wget ftp://ftp.uu.net/graphics/jpeg/jpegsrc.v6b.tar.gz
tar zxvf jpegsrc.v6b.tar.gz
cd jpeg-6b && ./configure && make && sudo make install
Install libgd from source:
mkdir -p ~/src && cd ~/src
curl http://www.libgd.org/releases/gd-2.0.35.tar.gz > gd-2.0.35.tar.gz
tar zxvf gd-2.0.35.tar.gz
cd gd-2.0.35 && ./configure && make && sudo make install
### RVideo (0.9.4)
Currently we can't use `sudo gem install rvideo` since that installs 0.9.3.
svn checkout svn://rubyforge.org/var/svn/rvideo/trunk rvideo
cd rvideo
rake install_gem
Install the rvideo tools (on OS X at least - your system may differ). You might want to check your gem library location (`gem env`).
sudo cp lib/rvideo/tools/*.rb /Library/Ruby/Gems/1.8/gems/rvideo-0.9.4/lib/rvideo/tools/.
### FFMPEG
Available in all good package repositories including Darwin Ports.
If you're developing on Mac OS X, you can save some time by [grabbing ffmpeg out of the ffmpegX application instead of compiling it](http://www.macosxhints.com/article.php?story=20061220082125312).
Grab Panda
----------
Get the latest (at this moment, our code is compatible with 0.9.7)
git clone git://github.com/newbamboo/panda.git
Development work is merged regularly into the master branch. If you have difficulty running try the stable branch which tracks releases.
You should follow the [getting started guide](http://pandastream.com/docs/getting_started#configure_panda) from the configure panda section onwards. You'll probably want to use the [filesystem storage option](http://pandastream.tumblr.com/post/54322685/panda-1-2-released) and also [emulate SimpleDB locally](http://pandastream.tumblr.com/post/52779609/playing-with-panda-without-simpledb-account).
Further information
===================
Investigating encoding errors
-----------------------------
When an encoding fails the status of the video is set to 'error' and the output of ffmpeg is saved on S3 with the filename `video_token.error`.
SimpleDB
========
Please refer to [amazon\_sdb](http://nytimes.rubyforge.org/amazon_sdb/) for how to access simpledb. Here are some examples.
Extracting All info
>> Profile.query
Extracting specific info
>> Profile.query("['audio_coded' = 'aac']")
Deleting a profile.
>> profile = Profile.query("['title'='Flash h264 SD']")
>> profile.destroy!
Videos schema
-------------
filename # 976a4b00-16cc-012b-7316-001ec2b5c0e1.flv
original_filename # sneezing_panda.flv
parent
status # original, queued, processing, done, error
duration
container
width
height
video_codec
video_bitrate
fps
audio_codec
audio_sample_rate
profile # id of encoding profile used
profile_title
updated_at
created_at
Encoding profiles schema
------------------------
title
container
width
height
video_codec
video_bitrate
fps
audio_codec
audio_bitrate
audio_sample_rate
updated_at
created_at

Просмотреть файл

@ -1,106 +0,0 @@
Panda on EC2
============
Launch the AMI
--------------
ec2-run-instances ami-468e6a2f -k pv-keypair
Nginx
-----
Required for progress uploads. Config is in /usr/local/nginx/conf/nginx.conf
Bin is /usr/local/nginx/sbin/nginx
Grab Panda
----------
cd /var/local/www
git clone git://github.com/newbamboo/panda.git
mkdir /var/local/www/panda/videos # Place for temp videos
mkdir /var/local/www/panda/log
Configure Panda
---------------
Copy the example config file and enter your AWS keys and other details.
cd /var/local/www/panda/config
cp panda_init.rb.example panda_init.rb
vi panda_init.rb
Create buckets, domains and queues
----------------------------------
* S3 bucket for video files (tip: name this a domain such videos.pandastream.com and you can setup a CNAME to S3)
* SimpleDB domains for videos, profiles and admin users
Login to the console with `merb -i` and run the following, probably replacing names with your own:
Panda::Setup.create_s3_bucket 'videos.pandastream.com'
Panda::Setup.create_sdb_domain 'panda_videos'
Panda::Setup.create_sdb_domain 'panda_profiles'
Panda::Setup.create_sdb_domain 'panda_users'
S3VideoObject.store("flvplayer.swf", File.open("flvplayer.swf"), :access => :public_read)
S3VideoObject.store('swfobject.js', open('public/javascripts/swfobject.js'), :access => :public_read)
Custom player and swfobject2
S3VideoObject.store('player.swf', open('public/player.swf'), :access => :public_read)
S3VideoObject.store('kleur.swf', open('public/kleur.swf'), :access => :public_read)
S3VideoObject.store('swfobject2.js', open('public/javascripts/swfobject2.js'), :access => :public_read)
S3VideoObject.store('expressInstall.swf', open('public/expressInstall.swf'), :access => :public_read)
Create first admin user
-----------------------
Login to the console with `merb -i`:
u = User.new
u.login = 'admin'
u.email = 'email@mydomain'
u.set_password('SECRETPASS')
u.save
Create an encoding profile
--------------------------
Chose one or more of the following. Uploaded videos will be encoded to all profiles.
Profile.create!(:title => "Flash video SD", :container => "flv", :video_bitrate => 300, :audio_bitrate => 48, :width => 320, :height => 240, :fps => 24, :position => 0, :player => "flash")
Profile.create!(:title => "Flash video HI", :container => "flv", :video_bitrate => 400, :audio_bitrate => 48, :width => 480, :height => 360, :fps => 24, :position => 1, :player => "flash")
Profile.create!(:title => "Flash h264 SD", :container => "mp4", :video_bitrate => 300, :audio_codec => "aac", :audio_bitrate => 48, :width => 320, :height => 240, :fps => 24, :position => 2, :player => "flash")
Profile.create!(:title => "Flash h264 HI", :container => "mp4", :video_bitrate => 400, :audio_bitrate => 48, :audio_coded => "aac", :width => 480, :height => 360, :fps => 24, :position => 3, :player => "flash")
Profile.create!(:title => "Flash h264 480p", :container => "mp4", :video_bitrate => 600, :audio_bitrate => 48, :audio_coded => "aac", :width => 852, :height => 480, :fps => 24, :position => 4, :player => "flash")
Configure Nginx
---------------
Edit your Nginx config as needed. The default config symlinks to `/usr/local/nginx/conf/nginx-panda.conf` and is setup to talk to Merb on port 4001.
/etc/init.d/nginx restart
Start Panda
-----------
You can start Merb directly:
cd /var/local/www/panda
merb -p 4001
The encoder can be run with
cd /var/local/www/panda
merb -r lib/encoder.rb -p 5001 -e encoder
Using God with the supplied config is the best method to control Merb and the encoder:
god
god load /var/local/www/panda/panda.god
Upload some videos!
-------------------
Visit http://ec2-instance-name-123-456.compute-1.amazonaws.com and login with the detail of the user you created earlier!

Просмотреть файл

@ -25,4 +25,9 @@ private
throw :halt, render('', :status => 401)
end
end
def swfobject2_link
"<script type=\"text/javascript\" charset=\"utf-8\" src= \"#{request.protocol}://#{request.host}/javascripts/swfobject2.js\"><\/script>"
end
end

Просмотреть файл

@ -38,5 +38,8 @@
<div class="embed_code">
<h4>Embed code</h4>
<textarea cols="50" rows="6"><%= @video.embed_js %></textarea>
<textarea cols="50" rows="6">
<%= swfobject2_link %>
<%= @video.embed_js %>
</textarea>
</div>

Просмотреть файл

@ -27,7 +27,6 @@
<dd><%= @video.duration_str %></dd>
</dl>
<% @video.encodings.each do |enc| %>
<h3><a href="/videos/<%= enc.key %>"><%= enc.profile_title %></a></h3>
@ -40,9 +39,10 @@
<div class="response">Once uploaded and encoded the video will be displayed here.</div>
<% end %>
</div>
<div class="embed_code">
<h4>Embed code</h4>
<textarea cols="50" rows="6"><%= enc.embed_js %></textarea>
<textarea cols="50" rows="6">
<%= swfobject2_link %>
<%= enc.embed_js %></textarea>
</div>
<% end %>

Просмотреть файл

@ -1,4 +1,4 @@
# merb -r "panda/lib/encoder.rb"
# merb -r "panda/bin/encoder.rb"
Merb.logger.info 'Encoder awake!'

Просмотреть файл

@ -1,4 +1,4 @@
# merb -r "panda/lib/notifier.rb"
# merb -r "panda/bin/notifier.rb"
# How notifications work:
# Once a video has been encoded its next_notification field will be set to the current time. It will then be returned by Video.outstanding_notifications and picked up by the notifier, which will then call send_notification on its parent.

Просмотреть файл

@ -2,50 +2,39 @@
Gem.clear_paths
Gem.path.unshift(Merb.root / "gems")
# Make the app's "lib" directory a place where ruby files get "require"d from
# Autoload from lib
$LOAD_PATH.unshift(Merb.root / "lib")
Merb.push_path(:lib, Merb.root / "lib") # uses **/*.rb as path glob.
Merb::Config.use do |c|
### Sets up a custom session id key, if you want to piggyback sessions of other applications
### with the cookie session store. If not specified, defaults to '_session_id'.
# c[:session_id_key] = '_session_id'
c[:session_id_key] = 'panda'
c[:session_secret_key] = '4d5e9b90d9e92c236a2300d718059aef3a9b9cbe'
c[:session_store] = 'cookie'
end
# use_orm :activerecord
# Load Panda config
require "config" / "panda_init"
dependencies 'merb-assets', 'merb-mailer', 'merb_helpers', 'uuid', 'to_simple_xml', 'rog', 'amazon_sdb', 'simple_db', 'retryable', 'activesupport', 'rvideo', 'panda', 'gd_resize', 'map_to_hash', 'spec_eql_hash', 'error_sender'
# Gem dependencies
dependency 'merb-assets'
dependency 'merb-mailer'
dependency 'merb_helpers'
dependency 'uuid'
dependency 'amazon_sdb'
dependency 'activesupport'
dependency 'rvideo'
dependencies 'abstract_store', 's3_store', 'file_store'
dependencies 'local_store'
# Not sure why dependencies won't load AWS::S3
require 'aws/s3'
require 'inline'
# Dependencies in lib - not autoloaded in time so require them explicitly
require 'simple_db'
require 'local_store'
# Check panda config
Panda::Config.check
Merb::BootLoader.after_app_loads do
# Panda specific
unless Merb.environment == "test"
require "config" / "aws"
require "config" / "mailer" # If you want notification and encoding errors to be sent to you as well as logged
end
# Overriding form, as SimpleDB does not provide errors on object.
module Merb::Helpers::Form
def _singleton_form_context
self._default_builder = Merb::Helpers::Form::Builder::ResourcefulForm
@_singleton_form_context ||=
self._default_builder.new(nil, nil, self)
end
require "config" / "mailer"
end
Store = case Panda::Config[:videos_store]
@ -65,5 +54,3 @@ Merb::BootLoader.after_app_loads do
Merb.logger.info "PANDA WARNING: Profile simple db domain does not exist. Please check that you have created all the required domains (see the getting started guide)."
end
end
EMAIL_SENDER = "Panda <info@pandastream.com>"

Просмотреть файл

Просмотреть файл

Просмотреть файл

@ -0,0 +1,28 @@
module Kernel
# Options
# =======
# * :tries Number of retries to perform. Defaults to 1.
# Note this means it actually tries 2 times!
# * :on - The Exception on which a retry will be performed. Defaults to
# Exception, which retries on any Exception.
#
# Example
# =======
# retryable(:tries => 1, :on => OpenURI::HTTPError) do
# # your code here
# end
#
def retryable(options = {}, &block)
opts = { :tries => 1, :on => Exception }.merge(options)
retry_exception, retries = opts[:on], opts[:tries]
begin
return yield
rescue retry_exception
retry if (retries -= 1) > 0
end
yield
end
end

Просмотреть файл

@ -1,3 +1,5 @@
require 'inline'
class GDResize
SUPPORTED_FORMATS = %w(jpg jpeg png gif)

8
lib/merb_helpers_form.rb Normal file
Просмотреть файл

@ -0,0 +1,8 @@
# Overriding form, as SimpleDB does not provide errors on object.
module Merb::Helpers::Form
def _singleton_form_context
self._default_builder = Merb::Helpers::Form::Builder::ResourcefulForm
@_singleton_form_context ||=
self._default_builder.new(nil, nil, self)
end
end

Просмотреть файл

@ -4,7 +4,7 @@ module Panda
class Setup
class << self
def create_s3_bucket
AWS::S3::Bucket.create(Panda::Config[:s3_videos_bucket])
S3Store.create_bucket
end
def create_sdb_domain(name)

Просмотреть файл

@ -1,23 +0,0 @@
# Options:
# * :tries - Number of retries to perform. Defaults to 1.
# * :on - The Exception on which a retry will be performed. Defaults to Exception, which retries on any Exception.
#
# Example
# =======
# retryable(:tries => 1, :on => OpenURI::HTTPError) do
# # your code here
# end
#
def retryable(options = {}, &block)
opts = { :tries => 1, :on => Exception }.merge(options)
retry_exception, retries = opts[:on], opts[:tries]
begin
return yield
rescue retry_exception
retry if (retries -= 1) > 0
end
yield
end

Просмотреть файл

@ -1,36 +0,0 @@
# == Synopsis
#
# Simple remote debug class
#
# == Author
# Stefan Saasen s@juretta.com
#
# == Copyright
# Copyright (c) 2005 juretta.com Stefan Saasen
# Licensed under the same terms as Ruby.
# == Version
# Version 0.1 ($Id: logger.rb 5 2006-01-01 12:51:04Z stefan $)
require 'socket'
require 'singleton'
require 'timeout'
class Rog
include Singleton
cattr_writer :port, :host, :prefix
attr :session
def self.log(level, msg)
begin
Timeout::timeout(1) do
@session = TCPSocket.new(@@host, @@port)
@session.puts Time.new.strftime("%Y-%m-%d %H:%M:%S") + \
" " + "[" + level.to_s.upcase + "] #{@@prefix}: " + msg + "\n"
@session.close
end
rescue => e
return false
end
true
end
end

Просмотреть файл

@ -1,3 +1,5 @@
require 'aws/s3'
class S3VideoObject < AWS::S3::S3Object
set_current_bucket_to Panda::Config[:s3_videos_bucket]
end
@ -13,6 +15,10 @@ class S3Store < AbstractStore
)
end
def self.create_bucket
AWS::S3::Bucket.create(Panda::Config[:s3_videos_bucket])
end
# Set file. Returns true if success.
def set(key, tmp_file)
begin

Просмотреть файл

@ -35,7 +35,7 @@ God.watch do |w|
w.name = "encoder"
current_path = "/var/local/www/panda"
port = 4091
w.start = "/bin/bash -c 'cd #{current_path}; merb -r lib/encoder.rb -d -p #{port} -e encoder'"
w.start = "/bin/bash -c 'cd #{current_path}; merb -r bin/encoder.rb -d -p #{port} -e encoder'"
w.stop = "/bin/bash -c 'cd #{current_path}; merb -k #{port}'"
w.pid_file = File.join(current_path, "log/merb.#{port}.pid")
w.behavior(:clean_pid_file)
@ -55,7 +55,7 @@ God.watch do |w|
w.name = "notifier"
current_path = "/var/local/www/panda"
port = 6001
w.start = "/bin/bash -c 'cd #{current_path}; merb -r lib/notifier.rb -d -p #{port} -e notifier'"
w.start = "/bin/bash -c 'cd #{current_path}; merb -r bin/notifier.rb -d -p #{port} -e notifier'"
w.stop = "/bin/bash -c 'cd #{current_path}; merb -k #{port}'"
w.pid_file = File.join(current_path, "log/merb.#{port}.pid")
w.behavior(:clean_pid_file)