a74e8273fd | ||
---|---|---|
cfg | ||
documents | ||
eng | ||
media | ||
modelOnnx | ||
src | ||
.gitattributes | ||
.gitignore | ||
CONTRIBUTING.md | ||
Config.bat | ||
LICENSE | ||
README.md | ||
SECURITY.md | ||
ThirdPartyNotices.md |
README.md
Microsoft Rocket Video Analytics Platform
A highly extensible software stack to empower everyone to build practical real-world live video analytics applications for object detection and counting/alerting with cutting edge machine learning algorithms. The repository features a hybrid edge-cloud video analytics pipeline (built on C# .NET Core), which allows TensorFlow DNN model plug-in, GPU/FPGA acceleration, docker containerization/Kubernetes orchestration, and interactive querying for after-the-fact analysis. A brief summary of Rocket platform can be found inside 📝Rocket-features-and-pipelines.pdf.
Feel free to check out our 📝webinar on Rocket from Dec 2019.
How to run the code
Step 1: Set up environment
Setup on Windows
-
Microsoft Visual Studio (VS 2017 is preferred) is recommended IDE for Rocket on Windows 10. While installing Visual Studio, please also add C++ 2015.3 v14.00 (v140) toolset to your local machine. Snapshot below shows how to include C++ 2015.3 v14.00 from Visual Studio Installer.
-
Follow instructions to install .NET Core 2.2 (2.2.102 is preferred).
-
To enable GPU support, install CUDA Toolkit and cuDNN. Please also make sure your NVIDIA driver is up-to-date.
-
CUDA 8.0 (e.g., cuda_8.0.61_win10_network.exe) is needed for Darknet (e.g., YOLO) models.
After installation, please make sure files in
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\extras\visual_studio_integration\MSBuildExtensions
are copied toC:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\BuildCustomizations
-
CUDA 9.1 (e.g., cuda_9.1.85_win10_network.exe) is needed to support TensorFlow models.
-
cuDNN v7 is preferred (e.g., cudnn-8.0-windows10-x64-v7.2.1.38.zip).
Copy
<installpath>\cuda\bin\cudnn64_7.dll
toC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\bin
.
Copy<installpath>\cuda\ include\cudnn.h
toC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include
.
Copy<installpath>\cuda\lib\x64\cudnn.lib
toC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\lib\x64
.
Add Variable Name:CUDA_PATH
with Variable Value:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0
into Environment Variables.
-
Restart your computer after installing CUDA and cuDNN.
-
Setup on Linux
Docker is recommended to run Rocket on Linux. Below we use Ubuntu 16.04 as an example to walk through the steps of building Rocket docker image and run it with GPU acceleration.
- Install .NET Core 2.2 SDK (2.2.301 is preferred).
- Install docker-ce (version 18.09.7 is preferred).
- Install NVIDIA driver based on your GPU model (e.g., 418.67 for Tesla GPU).
- Install nvidia-docker2. The NVIDIA Container Toolkit allows users to build and run GPU accelerated Docker containers. Note that you do NOT need to install the CUDA toolkit on the host, but GPU driver needs to be installed.
Step 2: Run the pipeline
Check out the repository.
Prepare video feeds and line configuration
- Prepare video feeds. Rocket can be fed with either live video streams (e.g.,
rtsp://<url>:<port>/
) or local video files (should be put into\media\
). A sample video filesample.mp4
is already included in\media\
. - Prepare a configuration file (should be placed into
\cfg\
) used in line-based counting/alerting and cascaded DNN calls. Each line in the file defines a line-of-interest with the format below.
<line_name> <line_id> <x_1> <y_1> <x_2> <y_2> <overlap_threshold>
A line configuration filesample.txt
manually created based onsample.mp4
is also included in the folder\cfg\
.
To learn more about configuring your own lines of interest, please visit this page.
Build on Windows
-
Run
Config.bat
before the first time you run Rocket to download pre-compiled OpenCV and TensorFlow binaries as well as Darknet YOLO weights files. It may take few minutes depending on your network status. Proceed only when all downloads finish. YOLOv3 and Tiny YOLOv3 are already included in Rocket. You can plug-in other YOLO models as you wish. -
Launch
VAP.sln
insrc\VAP\
from Visual Studio. -
Set pipeline config
PplConfig
in VideoPipelineCore - App.config. We have pre-compiled six configurations in the code. Pipeline descriptions are also included in 📝Rocket-features-and-pipelines.pdf.- 0: Line-based counting
- 1: Darknet Yolo v3 on every frame (slide #7)
- 2: TensorFlow FastRCNN on every frame (slide #8)
- 3: Background subtraction-based (BGS) early filtering -> Darknet Tiny Yolo -> Darknet Yolo v3 (slide #9)
- 4: BGS early filtering -> Darknet Tiny Yolo -> Database (ArangoDB and blob storage on Azure) (slide #10)
- 5: BGS early filtering -> TensorFlow Fast R-CNN -> Azure Machine Learning (cloud) (slide #11)
-
(Optional) Set up your own database and Azure Machine Learning service if
PplConfig
is set to 4 or 5.- Azure Database:
- Deploy SQL database like MySQL or NoSQL database such as ArangoDB on Azure by creating a VM.
- Supply database settings (e.g., server name, user name, credentials etc.) to Rocket in
App.Config
. - You can also set up your cloud storage (e.g., Azure Blob Storage) to store images/videos. In pipeline 4, Rocket sends detection images to an Azure storage account and metadata to an Azure database.
- Azure Machine Learning:
- Deploy your deep learning models to Azure (e.g., using Azure Kubernetes Service or AKS) for inference with GPU or FPGA.
- After deploying your model as a web service, provide host URL, key, and service ID to VAP in
App.Config
. Rocker will handle the communication between local modules and the cloud service.
- Azure Database:
-
Build the solution.
-
Run the code.
- Using Visual Studio: set VideoPipelineCore - Property - Debug - Application Arguments
<video_file/camera_url> <line_detection_config_file> <sampling_factor> <resolution_factor> <object_category>
. To run Rocket on the sample video, for example, arguments can be set tosample.mp4 sample.txt 1 1 car
. - Using Command Line (CMD or PowerShell): run
dotnet .\VideoPipelineCore.dll <video_file/camera_url> <line_detection_config_file> <sampling_factor> <resolution_factor> <object_category>
in\src\VAP\VideoPipelineCore\bin\Debug\netcoreapp2.2
. For instance,dotnet .\VideoPipelineCore.dll sample.mp4 sample.txt 1 1 car
.
- Using Visual Studio: set VideoPipelineCore - Property - Debug - Application Arguments
Build on Linux
We have pre-built a Rocket docker image from docker branch with local processing only (slide #12 without cloud parts). The image is hosted on Docker Hub, a public library and community for container images, and you will be asked to login before pull/push images (sign up first if you don't have an account).
To test on the pre-built Rocket image, run
docker pull ycshu086/rocket-sample-edgeonly:0.1
Once pulled, run the command below to start Rocket with NVIDIA GPU.
docker run --runtime=nvidia -v <local directory>:/app/output ycshu086/rocket-sample-edgeonly:0.1 sample.mp4 sample.txt 1 1 car
-
Build your own Rocket pipeline on Linux
- Pull base docker image with CUDA toolkit and OpenCV. This image is needed to build Rocket docker image.
docker pull ycshu086/ubuntu-dotnetcore-opencv-opencvsharp-cuda-cudnn:<version>
. - Git clone docker branch for source code to dockerize Rocket on Linux.
- Create line configuration file(s) inside
\cfg
. If you are running Rocket on a pre-recorded video, please also copy the video file into\media
. - (Optional) Update
\src\VAP\VideoPipelineCore\App.Config
to set proper parameters for database and Azure Machine Learning service connection. - Run
sudo chmod 744 Config.sh
andsudo ./Config.sh
before the first time you build Rocket image to download pre-compiled TensorFlow binaries. - Run
docker build
to build Rocket image usingDockerfile.VAP
.
docker build -t <repository>/<image>:<version> -f Dockerfile.VAP .
- (Optional) Push Rocket image to a cloud repository (e.g., docker hub, Azure Container Registry etc.) if you need to run it somewhere else.
docker push -t <repository>/<image>:<version>
- Pull base docker image with CUDA toolkit and OpenCV. This image is needed to build Rocket docker image.
-
Run Rocket image on Linux
-
Pull a pre-built Rocket docker image to the local machine. You can use
docker images
to check existing images.
docker pull -t <repository>/<image>:<version>
-
Mount volume into the container and run Rocket image with NVIDIA GPU.
docker run --runtime=nvidia -v <local directory>:/app/output <repository>/<image>:<version> sample.mp4 sample.txt 1 1 car
-
Step 3: Results
Output images are generated in folders in \src\VAP\VideoPipelineCore\bin\
(Windows), or the local directory you mount during docker run
on Linux. Results from different modules are sent to different directories (e.g., output_bgsline
for background subtraction-based detector) whereas output_all
has images from all modules. Name of each file consists of frame ID, module name, and confidence score. Below are few sample results from running pipeline 3 and pipeline 5 on sample.mp4
. You should also see results printed in console during running.
The above illustration on pipeline 3 shows that at frame 2679, background subtraction detected an object, tiny Yolo DNN confirmed it was a car with a confidence of 0.24, and heavy Yolo v3 confirmed it with a confidence of 0.92. Likewise, for pipeline 5 where the TensorFlow FastRNN model had a confidence of 0.55 and AzureML (in the cloud) came back with a confidence of 0.76 for the same object.