MixedRealityCompanionKit/LegacySpectatorView

NOTE: The Spectator View code within this repository has been deprecated. To use Spectator View, migrate to the MixedReality-SpectatorView repository.

README

These instructions are for the UNET implementation of spectator view For the HoloToolkit sharing service implementation, see the Legacy Documentation.

Overview

Spectator view uses an external camera rigidly mounted to a HoloLens to take high definition pictures and video of your holographic Unity project. Unlike Mixed Reality Capture (MRC), the Spectator View project allows for high resolution captures from a third person perspective Use this project to capture higher resolution and higher quality pictures for your store submissions, marketing videos, or for projecting a live view for all spectators to see.

DISCLAIMER: Video recording is in alpha and you might experience some latency between hologram position in video to real-life. Move the camera slower or target a stationary camera to mitigate this.

Material List

Note: The linked materials are not the only materials that will work, but are the ones that we used. Make sure the materials you get are compatible with your existing camera.

• Camera with HDMI out and configurable output overlays.
  • See below for tested cameras.
• Capture Card
  • See below for tested capture cards.
• HDMI to Mini HDMI cable (or whatever HDMI form your camera supports)
• Bracket to connect your HoloLens to a camera: Hardware\HoloLens_Mount.stp.
  • We recommend this to be professionally machined in aluminum: 3D printing might not achieve adequate tolerances to hold the device.
    • Search for machine shops in your area to get this bracket machined.
  • You do not have to use this bracket: Any means of rigidly attaching your HoloLens to your camera will work.
• If using the above mount:
  • Adapter to connect the HoloLens mount to a DSLR hotshoe: Hardware\Mount_Adapter.stl. This file can be 3D printed. Holes were measured against the below bolts, you might need to alter the dimensions if using other bolts.
  • Hotshoe Fastener
  • 1/4-20" x 3/4" Bolts
  • 1/4-20" Nuts
  • Screw driver with the following heads (These links are examples, many screwdriver sets will have these attachments) :
    • 7/16 Nut Driver
    • T15 Torx
    • T7 Torx
• Router
  • This is necessary if your network does not support UDP broadcast.

Software

• Blackmagic Capture Card SDK
  • Search for Desktop Video SDK in "Latest Downloads"
  • If using a BlackMagic DeckLink card.
• Blackmagic Desktop Video Runtime
  • Search for Desktop Video Update in "Latest Downloads"
  • Version number must match the Capture Card SDK.
  • If using a BlackMagic DeckLink card.
• Elgato Game Capture HD
  • If using an Elgato capture card.
• Elgato Filters
  • If using an Elgato capture card.
• OpenCV 3.2
  • Download and install OpenCV 3.2
• Canon SDK (Optional)
  • If you are using a Canon camera and have access to the Canon EDSDK, you can tether your camera to your PC to take higher resolution images.
• Unity3D
  • UNET implementation works best in Unity 5.6.1+
• Visual Studio 2017
  • Note: Visual Studio 2015 may also be used, but the solutions will need to be rolled back to v140 and you will need to target an earlier UWP SDK (such as 10586 or 14393)
  • Ensure you have the following VS2017 Workloads installed: 
    1. Universal Windows Platform development 
    2. Desktop development with C++ 
    3. Game development with C++
• Windows Media Feature Pack
  • Only required for Windows N or KN editions.

Dependencies

The following dependencies require a manual download and a Visual Studio user macro.

OpenCV 3.2

• Download and install OpenCV 3.2 from here.
• Extract OpenCV anywhere on your computer.
• Update the OpenCV_vc14 user macro in dependencies.props with the corresponding path on your computer.
• Restart Visual Studio if the Calibration or Compositor sln's are open.

DeckLink Capture Card

• Download the DeckLink SDK from here.
  • Search for Desktop Video SDK in "Latest Downloads"
• Extract the SDK anywhere on your computer.
• Update the DeckLink_inc user macro in dependencies.props with the corresponding path on your computer.
• Restart Visual Studio if the Calibration or Compositor sln's are open.

Elgato Capture Card

• Download the Elgato DirectShow filter headers from here.
• Clone or extract anywhere on your computer.
• Update the Elgato_Filter user macro in dependencies.props with the corresponding path on your computer.
• Restart Visual Studio if the Calibration or Compositor sln's are open.

Canon SDK (Optional)

• Request access to the Canon SDK from here.
• Extract anywhere on your computer.
• Update the Canon_SDK user macro in dependencies.props with the corresponding path on your computer.
• NOTE: When targetting x64 Unity, use EDSDK_64. When targetting x86 Unity, use EDSDK.
• Restart Visual Studio if the Calibration or Compositor sln's are open.

Included Projects

Each project has its own README with additional overview and setup instructions. See Documentation links below.

Calibration

Calibrates the HoloLens to the external camera. Uses a checkerboard pattern to find the rotation and translation from the HoloLens to the external camera. This process also finds the vertical field of view of the attached camera so holograms can be rendered at the correct projection to the camera.

Compositor

Unity Editor window that renders holograms from Unity over a color frame from a capture card. Outputs photo and video of the scene. This must be built and copied to your Unity project using CopyDLL.cmd.

Samples\SharedHolograms

Sample demonstrating how to add the HolographicCameraRig to an app that is already a shared experience. In this case, it is a simple shared experience using UNET.

CopyDLL.cmd

If you have changed any #define in CompositorShared.h, run this cmd to copy DLL's that depend on those definitions to your Unity projects. Include the Assets path to your own project to copy there.

Camera Setup

If using a camera with a capture card, you must configure the camera to output correctly:

• Change your camera to Video Mode. This will allow the output image to fill the whole buffer, rather than letterboxing to the picture aspect ratio.
• Find your camera's HDMI settings - enable "dual monitor" or "mirroring".
• Set the resolution to 1080P
• Turn off "Live view on screen display" so any screen overlays on the live view are not shown in the composite.
• Turn on your camera's live view.
• If using the Canon SDK and would like to use a flash unit, disable "Silent LV Shoot"
• Plug an hdmi cable from the camera's hdmi port to the hdmi in port of the capture card.
  • This is the outer port on the Blackmagic capture card.
• Optionally plug an hdmi cable from the hdmi out port to a television or preview monitor.
  • This is the center port on the Blackmagic capture card.

Note: if you would like to capture images or video at a resolution other than 1080P (eg: Photo Mode, 720P, or 4K) there will be a small code change mentioned in the individual projects' README's.

Network Setup

• The networking stack requires UDP broadcast to be enabled on your network.
• If you are on a corporate network that does not have this enabled, you may need to use a router
  • Plug an ethernet cable from your PC into the router.
  • Connect your HoloLens devices to this router.
• Ensure your HoloLens devices and Unity can talk to each other by opening up UDP and TCP communication to Unity in your firewall settings.
  • Open the Control Panel.
  • Select System and Security.
  • Select Windows Firewall.
  • Select Advanced Settings.
  • Select Inbound Rules.
  • Find Unity Editor and ensure the Program path is the version of Unity you are using.
  • Double click on the Unity Editor line with the UDP protocol.
  • Select Allow the connection.
  • Select Advanced.
  • Select the profiles that will work for your router. (At least Private must be checked when using a router.)
  • Select Apply.
  • Repeat for the TCP protocol.

Spectator View Rig Setup

This section assumes you are using the included HoloLens bracket. Alternative mounting techniques can be used instead.

• Use a T7 screwdriver to remove the headband from the HoloLens.
  • Once screws are loose, poke them out with a paperclip from the other side.
• Remove the screw cap on the inside front of the HoloLens visor, you can do this with a small flat head screwdriver.

• Use a T15 screwdriver to remove the small torx bolts from the HoloLens bracket to remove the U and Hook-shaped attachments.
• Place the HoloLens on the bracket, lining up the exposed hole on the inside of the visor with the extrusion on the front of the bracket.
  • HoloLens arms should be kept in place by the pins on the bottom of the bracket.
• Reattach the U and Hook-shaped attachments to secure the HoloLens to the bracket.

• Attach the hotshoe fastener to the hotshoe of your camera.
• Attach the mount adapter to the hotshoe fastener.

• Rotate the adapter so the narrow side is facing forward and parallel to the camera's lens.
• Secure the adapter in place with a 1/4" nut using the 7/16 nut driver.
• Position the bracket against the adapter so the front of the HoloLens' visor is as close as possible to the front of the camera's lens.
• Attach the bracket with 4 1/4" nuts and bolts using the 7/16 nut driver.

Alternate Spectator View Rig Setup

The spectator view rig assembly can be made without any custom hardware.

Material List

• 1 Hotshoe Fastener
• 1 Dual Flash Bracket
• 1 Washer
• 2 Clamps
• 2 6/32 Bolts
• 2 6/32 Nuts
• Cardboard or stiff foam

Assembly

• Attach the hotshoe fastener to the camera's hotshoe using one of the included nuts.
• Place a washer on the hotshoe fastener above the nut.
• Attach the dual flash bracket to the hotshoe fastener.
  • The washer should guarantee that the bracket is perpendicular to the lens.
• Remove the included thumbscrews from the dual flash bracket.
• Attach the clamps to each side of the dual flash bracket with the 6/32 nuts and bolts.
  • Ensure the arms of the HoloLens will be able to fit in the clamps with the visor centered on the camera's lens.
• Place each HoloLens arm in the open clamps.
• Place some cardboard or stiff foam in between the headband and the arms to act as a buffer against the clamp's tension.
• Without overtightening, tighten the clamps to keep the HoloLens in place.

Recommended PC Specs

Any PC that can run the HoloLens emulator can run spectator view.

Tested Cameras

• Canon 5D Mark III
• Nikon D7200
• GoPro Hero 4 Black
• Blackmagic Design Production Camera 4K with SDI to HDMI converter

Tested Capture Cards

• Blackmagic Intensity Pro 4K
• Blackmagic Intensity Shuttle USB 3
  • Does not work on all PCs - see Blackmagic documentation for their tested hardware.
  • Only supports HDMI in from camera (no HDMI out to an external display).
• Elgato HD 60S
  • Only supports HDMI in from camera (no HDMI out to an external display).

NOTE: The default capture card will be a blackmagic capture card, if using a different card, you will need to update Compositor\SharedHeaders\CompositorShared.h:

• USE_DECKLINK is TRUE if using a Blackmagic card.
• USE_ELGATO is TRUE if using an Elgato card.
• USE_OPENCV is TRUE to fallback to any other camera.

Each capture card has required software dependencies including runtimes and SDK's. See the Software section above.

Adding Spectator View Support to Your Unity App

• If your application does not use the HoloToolkit, you must first add the HoloToolkit.
  • This can be done from the spectator view releases in the HoloLens Companion Kit repository for a minimum set of the HoloToolkit for spectator view compatibility.
  • Find the largest number subdirectory for the latest official release.
  • Import the HoloToolkit_Min_%number%.unitypackage
• Import the SpectatorView_%number%.unitypackage from releases, to add the spectator view code to your project.
• Add the "Addons\HolographicCameraRig\Prefabs\SpectatorViewManager" prefab to your Hierarchy.
• Add the "Addons\HolographicCameraRig\SV_UNET\Prefabs\Sharing prefab to your Hierarchy.
• Build "Compositor\Compositor.sln" for x86 and x64 and run CopyDLL.cmd with the Assets directory of your project as the only parameter.
  • For more information and build configuration options, read the compositor README.
• Restart Unity for DLLs to import.
• For more information, read the sample README.

Documentation

• Overview
• Calibration
• Compositor
• Sample