Merge changes from upstream toolboc/IntelligentEdgeHOL

.env

@@ -2,4 +2,4 @@ CONTAINER_REGISTRY_URL=toolboc
 CONTAINER_REGISTRY_USERNAME=
 CONTAINER_REGISTRY_PASSWORD=
 CONTAINER_MODULE_VERSION=latest
-CONTAINER_VIDEO_SOURCE=https://www.youtube.com/watch?v=YZkp0qBBmpw
+CONTAINER_VIDEO_SOURCE=https://www.youtube.com/watch?v=XJ735krOiPo

README.md

@@ -12,102 +12,34 @@ urlFragment: "https://github.com/Azure/IntelligentEdgeHOL"
 
 ![](https://pbs.twimg.com/media/D_ANZnbWsAA4EVK.jpg)
 
-The IntelligentEdgeHOL walks through the process of deploying an IoT Edge module to an Nvidia Jetson Nano device to allow for detection of objects in YouTube videos, RTSP streams, or an attached web cam. It achieves performance of around 10 frames per second for most video data.    
+The IntelligentEdgeHOL walks through the process of deploying an [IoT Edge](https://docs.microsoft.com/en-us/azure/iot-edge/about-iot-edge?WT.mc_id=github-IntelligentEdgeHOL-pdecarlo) module to an Nvidia Jetson Nano device to allow for detection of objects in YouTube videos, RTSP streams, Hololens Mixed Reality Capture, or an attached web cam. It achieves performance of around 10 frames per second for most video data.    
 
 The module ships as a fully self-contained docker image totalling around 5.5GB.  This image contains all necessary dependencies including the [Nvidia Linux for Tegra Drivers](https://developer.nvidia.com/embedded/linux-tegra) for Jetson Nano, [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit), [NVIDIA CUDA Deep Neural Network library (CUDNN)](https://developer.nvidia.com/cudnn), [OpenCV](https://github.com/opencv/opencv), and [Darknet](https://github.com/AlexeyAB/darknet). For details on how the base images are built, see the included `docker` folder.
 
 Object Detection is accomplished using YOLOv3-tiny with [Darknet](https://github.com/AlexeyAB/darknet) which supports detection of the following:
 
-```
+
-person
+*person, bicycle, car, motorbike, aeroplane, bus, train, truck, boat, traffic light, fire hydrant, stop sign, parking meter, bench, bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe, backpack, umbrella, handbag, tie, suitcase, frisbee, skis, snowboard, sports ball, kite, baseball bat, baseball glove,skateboard, surfboard, tennis racket, bottle, wine glass, cup, fork, knife, spoon, bowl, banana, apple, sandwich, orange, broccoli, carrot, hot dog, pizza, donut, cake, chair, sofa, pottedplant, bed, diningtable, toilet, tv monitor, laptop, mouse, remote, keyboard, cell phone, microwave, oventoaster, sink, refrigerator, book, clock, vase, scissors, teddy bear, hair drier, toothbrush*
-bicycle
+
-car
+# Demos
-motorbike
+
-aeroplane
+* [Yolo Object Detection with Nvidia Jetson and Hololens](https://www.youtube.com/watch?v=zxGcUmcl1qo&feature=youtu.be)
-bus
+
-train
+# Hands-On Lab Materials 
-truck
+
-boat
+* [Presentation Deck](http://aka.ms/intelligentedgeholdeck)
-traffic light
+* [Presentation Video](http:aka.ms/intelligentedgeholvideo)
-fire hydrant
+  - Note: If you want to view a full walkthrough of this lab, skip to 38:00
-stop sign
+
-parking meter
-bench
-bird
-cat
-dog
-horse
-sheep
-cow
-elephant
-bear
-zebra
-giraffe
-backpack
-umbrella
-handbag
-tie
-suitcase
-frisbee
-skis
-snowboard
-sports ball
-kite
-baseball bat
-baseball glove
-skateboard
-surfboard
-tennis racket
-bottle
-wine glass
-cup
-fork
-knife
-spoon
-bowl
-banana
-apple
-sandwich
-orange
-broccoli
-carrot
-hot dog
-pizza
-donut
-cake
-chair
-sofa
-pottedplant
-bed
-diningtable
-toilet
-tvmonitor
-laptop
-mouse
-remote
-keyboard
-cell phone
-microwave
-oven
-toaster
-sink
-refrigerator
-book
-clock
-vase
-scissors
-teddy bear
-hair drier
-toothbrush
-```
 
 # Getting Started
 This lab requires that you have the following:
 
 Hardware:
 * [Nvidia Jetson Nano Device](https://amzn.to/2WFE5zF)
-* A cooling fan installed on or pointed at the Nvidia Jetson Nano device 
+* A [cooling fan](https://amzn.to/2ZI2ki9) installed on or pointed at the Nvidia Jetson Nano device 
-* USB Webcam (Optional)
+* USB Webcam (Optional) 
+  - Note: The power consumption will require that your device is configured to use a [5V/4A barrel adapter](https://amzn.to/32DFsTq) as mentioned [here](https://www.jetsonhacks.com/2019/04/10/jetson-nano-use-more-power/) with an [Open-CV compatible camera](https://web.archive.org/web/20120815172655/http://opencv.willowgarage.com/wiki/Welcome/OS/).
 
 Development Environment:
 - [Visual Studio Code (VSCode)](https://code.visualstudio.com/Download?WT.mc_id=github-IntelligentEdgeHOL-pdecarlo)
@@ -121,10 +53,10 @@ Development Environment:
 
 # Installing IoT Edge onto the Jetson Nano Device
 
-Before we install IoT Edge, we need to install a few utitilies onto the Nvidia Jetson Nano device with:
+Before we install IoT Edge, we need to install a few utilities onto the Nvidia Jetson Nano device with:
 
 ```
-apt-get install -y curl nano python3-pip
+sudo apt-get install -y curl nano python3-pip
 ```
 
 ARM64 builds of IoT Edge are currently being offered in preview and will eventually go into General Availability.  We will make use of the ARM64 builds to ensure that we get the best performance out of our IoT Edge solution.
@@ -163,7 +95,7 @@ Once you have obtained a connection string, open the configuration file:
 sudo nano /etc/iotedge/config.yaml
 ```
 
-Find the provisioning section of the file and uncomment the manual provisioning mode. Update the value of device_connection_string with the connection string from your IoT Edge device.
+Find the provisioning section of the file and uncomment the manual provisioning mode. Update the value of `device_connection_string` with the connection string from your IoT Edge device.
 
 ```
 provisioning:
@@ -178,6 +110,12 @@ provisioning:
 
 ```
 
+After you have updated the value of `device_connection_string`, restart the iotedge service with:
+
+```
+sudo service iotedge restart
+```
+
 You can check the status of the IoT Edge Daemon using:
 
 ```
@@ -207,6 +145,9 @@ In VS Code, navigate to the `.env` file and modify the following value:
 
  For an rtsp stream, provide a link to the rtsp stream in the format, rtsp://
 
+ To use a HoloLens video stream, see this [article](https://blog.kloud.com.au/2016/09/01/streaming-hololens-video-to-your-web-browser/) to enable a user account in the HoloLens Web Portal, once this is configured,  provide the url to the HoloLens video streaming endpoint, ex:
+ https://[USERNAME]:[PASSWORD]@[HOLOLENS_IP]/api/holographic/stream/live_high.mp4?holo=true&pv=true&mic=true&loopback=true
+
  If you have an attached USB web cam, provide the V4L device path (this can be obtained from the terminal with `ls -ltrh /dev/video*`, ex: /dev/video0 and open the included `deployment.template.json` and look for:
 
  ```
@@ -261,7 +202,7 @@ You can Open this Web Server using the IP Address or Host Name of the Nvidia Jet
 
 Example :
  
- http://JetsonNano  
+ http://jetson-nano-00  
  
  or 
  
@@ -319,20 +260,6 @@ WARNING: Assuming --restrict-filenames since file system encoding cannot encode
 Download Complete
 ```
 
-# Enable Object Detection by modifying the Module Twin
-
-While in VSCode, select the Azure IoT Hub Devices window, find your IoT Edge device and expand the modules sections until you see the `YoloModule` entry.
-
-Right click on `YoloModule` and select `Edit Module Twin`
-
-A new window name `azure-iot-module-twin.json` should open.
-
-Set the value of `properties -> desired -> Inference` to 1
-
-Right click anywhere in the Editor window, then select `Update Module Twin`
-
- After a few moments the object detection feature will become enabled in the module.  Now, if you reconnect to the video stream connected to in the previous step, you should see a bounding box and tags appearing around any detected objects in the video stream.
-
 # Monitor the GPU utilization stats
 
 On the Jetson device, you can monitor the GPU utilization by installing `jetson-stats` with:
@@ -374,7 +301,24 @@ Confidence Level threshold. The module ignores any inference results below this
 `VideoSource` : (string)
 Source of video stream/capture source
 
+# Pushing Detected Object Data into Azure Time Series Insights
 
+[Azure Time Series Insights](https://docs.microsoft.com/en-us/azure/time-series-insights/time-series-insights-overview?WT.mc_id=github-IntelligentEdgeHOL-pdecarlo) is built to store, visualize, and query large amounts of time series data, such as that generated by IoT devices. This service can allow us to extract insights that may allow us to build something very interesting. For example, imagine getting an alert when the mail truck is actually at the driveway, counting wildlife species using camera feeds from the National Park Service, or being able to tell that people are in a place that they should not be or counting them over time!
+
+To begin, navigate to the resource group that contains the IoT Hub that was created in the previous steps. Add a new Time Series Insights environment into the Resource Group and select `S1` tier for deployment.  Be sure to place the Time Series Insights instance into the same Geographical region which contains your IoT Hub to minimize latency and egress charges.
+
+![](https://hackster.imgix.net/uploads/attachments/939871/image_11Mggcf7p3.png?auto=compress)
+
+Next, choose a unique name for your Event Source and configure the Event Source to point to the IoT Hub you created in the previous steps. Set the `IoT Hub Access Policy Name` to "iothubowner", be sure to create a new IoT Hub Consumer Group named "tsi", and leave the `TimeStamp Propery Name` empty as shown below:
+
+![](https://hackster.imgix.net/uploads/attachments/939872/image_4DsJXUVxvt.png?auto=compress)
+
+Complete the steps to "Review and Create" your deployment of Time Series Insights.  Once the instance has finished deploying, you can navigate to the Time Insights Explorer by viewing the newly deployed Time Series Insights Environment resource, selecting "Overview" and clicking the "Time Series Insights explorer URL".  Once you have clicked the link, you may begin working with your detected object data.
+
+For details on how to explore and query your data in the Azure Time Series Insights explorer, you may consult the [Time Series Insights documentation](https://docs.microsoft.com/en-us/azure/time-series-insights/time-series-insights-explorer?WT.mc_id=github-IntelligentEdgeHOL-pdecarlo).
+
+![](https://hackster.imgix.net/uploads/attachments/939873/image_JWWcQszXsh.png?auto=compress)
+ 
 ## Contributing
 
 This project welcomes contributions and suggestions.  Most contributions require you to agree to a

config/deployment.arm32v7.json

@@ -43,7 +43,7 @@
             "restartPolicy": "always",
             "settings": {
               "image": "toolboc/yolomodule:latest-arm32v7",
-              "createOptions": "{\"Env\":[\"VIDEO_PATH=https://www.youtube.com/watch?v=YZkp0qBBmpw\",\"VIDEO_WIDTH=0\",\"VIDEO_HEIGHT=0\",\"FONT_SCALE=0.8\"],\"HostConfig\":{\"Devices\":[{\"PathOnHost\":\"/dev/nvhost-ctrl\",\"PathInContainer\":\"/dev/nvhost-ctrl\",\"CgroupPermissions\":\"rwm\"},{\"PathOnHost\":\"/dev/nvhost-ctrl-gpu\",\"PathInContainer\":\"dev/nvhost-ctrl-gpu\",\"CgroupPermissions\":\"rwm\"},{\"PathOnHost\":\"/dev/nvhost-prof-gpu\",\"PathInContainer\":\"dev/nvhost-prof-gpu \",\"CgroupPermissions\":\"rwm\"},{\"PathOnHost\":\"/dev/nvmap\",\"PathInContainer\":\"/dev/nvmap\",\"Cgroup",
+              "createOptions": "{\"Env\":[\"VIDEO_PATH=https://www.youtube.com/watch?v=XJ735krOiPo\",\"VIDEO_WIDTH=0\",\"VIDEO_HEIGHT=0\",\"FONT_SCALE=0.8\"],\"HostConfig\":{\"Devices\":[{\"PathOnHost\":\"/dev/nvhost-ctrl\",\"PathInContainer\":\"/dev/nvhost-ctrl\",\"CgroupPermissions\":\"rwm\"},{\"PathOnHost\":\"/dev/nvhost-ctrl-gpu\",\"PathInContainer\":\"dev/nvhost-ctrl-gpu\",\"CgroupPermissions\":\"rwm\"},{\"PathOnHost\":\"/dev/nvhost-prof-gpu\",\"PathInContainer\":\"dev/nvhost-prof-gpu \",\"CgroupPermissions\":\"rwm\"},{\"PathOnHost\":\"/dev/nvmap\",\"PathInContainer\":\"/dev/nvmap\",\"Cgroup",
               "createOptions01": "Permissions\":\"rwm\"},{\"PathOnHost\":\"dev/nvhost-gpu\",\"PathInContainer\":\"dev/nvhost-gpu\",\"CgroupPermissions\":\"rwm\"},{\"PathOnHost\":\"/dev/nvhost-as-gpu\",\"PathInContainer\":\"/dev/nvhost-as-gpu\",\"CgroupPermissions\":\"rwm\"},{\"PathOnHost\":\"/dev/nvhost-vic\",\"PathInContainer\":\"/dev/nvhost-vic\",\"CgroupPermissions\":\"rwm\"},{\"PathOnHost\":\"/dev/tegra_dc_ctrl\",\"PathInContainer\":\"/dev/tegra_dc_ctrl\",\"CgroupPermissions\":\"rwm\"}],\"PortBindings\":{\"80/tcp\":[{\"HostPort\":\"80\"}]}}}"
             }
           }

modules/YoloModule/Dockerfile.arm64v8

@@ -13,7 +13,7 @@ WORKDIR /usr/sdk
 RUN python -m virtualenv --python=python3 env3
 RUN source env3/bin/activate && pip install --upgrade pip && pip install -U setuptools wheel
 
-RUN git clone --recursive --depth=1 https://github.com/Azure/azure-iot-sdk-python.git src
+RUN git clone --recursive --branch release_2019_01_03 --depth=1 https://github.com/Azure/azure-iot-sdk-python.git src
 
 # Build for Python 3
 RUN add-apt-repository ppa:deadsnakes/ppa
@@ -43,8 +43,16 @@ RUN cp /usr/local/src/darknet/libdarknet.so /app/libdarknet.so
 COPY /build/requirements.txt ./
 RUN pip3 install --upgrade pip
 RUN pip3 install --no-cache-dir -r requirements.txt
-RUN pip3 install tornado==4.5.3 trollius && \
+RUN pip3 install tornado==4.5.3 trollius 
-    pip3 install -U youtube-dl
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends zip pandoc && \
+    rm -rf /var/lib/apt/lists/*
+
+RUN git clone --depth=1 https://github.com/ytdl-org/youtube-dl.git && \
+    cd youtube-dl && \
+    make && \
+    make install
 
 ADD /app/ .
 

modules/YoloModule/app/VideoCapture.py

@@ -70,7 +70,8 @@ class VideoCapture(object):
 
     def __IsRtsp(self, videoPath):
         try:
-            return 'rtsp:' in videoPath.lower()
+            if 'rtsp:' in videoPath.lower() or '/api/holographic/stream' in videoPath.lower():
+                return True
         except ValueError:
             return False