XRAutomatedTests/GraphicsTests

XR Graphics Tests

• GraphicsTests/LWRPSmokeTest - single simple scene that uses the lightweight scriptable render pipeline.
• GraphicsTests/LegacyRenderer - handful of tests from the legacy renderer test project in the Unity repo that runs on Katana. Tests were chosen pretty loosely from the first half of the tests in that project. There are about 80 tests total in this suite.
• GraphicsTests/LegacySmokeTest - single simple scene that uses the legacy renderer.
• GraphicsTests/LightweightRenderer - this is a copy of the test project from the scriptable render pipeline repository. Some tests filtered out due to multiple view ports not being supported. Hope to just run these tests out of the SRP repo in the future.

Graphics Test Framework Overview

All of the graphics test projects use the Graphics Test Framework which is a collection of scripts that generate Unity Test Framework tests during setup. When a Unity test is decorated with the UseGraphicsTestCases attribute the framework will search through the build settings for scenes, open the scene and check if it needs to filter the scene out and whether to bake lightmaps, does the appropriate action and then generates a test. A GraphicsTestCase object is passed in to the test which contains specific information for each test when it is run. This contains the scene name and reference image location so they can be referenced in the test.

See the Graphics Test Framework Readme for more information on the framework.

Running Tests locally

Most of the test projects use a setup method to configure the player settings passed in through the cmd line and needs to be called using the -executeMethod parameter. Most graphics test projects use a method called Setup.SetupAll to initiate this.

The projects use a configuration manager local package to set these and the setup method for that is called in the test setup script.

Player settings parameters that are currently configurable in graphics projects:

• -enabledxrtarget=[Oculus, OpenVR, cardboard, daydream, MockHMD, None] - This is the XR SDK to set. Defaults to "None"
• -playergraphicsapi=[UnityEngine.Rendering.GraphicsDeviceType] - The graphics api to use. Use string representations of the Unity enum for this. Defaults to whatever the project is set to in the settings.
• -colorspace=[Linear, Gamma] - The colorspace to use. Defaults to the player settings option.
• -stereorenderingpath=[Multipass, SinglePass, Instancing] - The stereo rendering path to use. Defaults to SinglePass.
• -mtrendering=[true, false] - Sets multithreading rendering on or off. Defaults to True.
• -graphicsjobs=[true, false] - Sets graphics jobs on or off. Defaults to false.
• -minimumandroidsdkversion=[UnityEditor.AndroidSdkVersions] - Minimum Android sdk version to use. Use string representation of the Unity enum. Defaults to the player settings.
• -targetandroidsdkversion=[UnityEditor.AndroidSdkVersions] - Target Android sdk version to use. Same usage as minimum version.

Example cmd line which would run the legacy graphics tests

Unity.exe -runTests -automated -projectPath xrautomatedtests\GraphicsTests\LegacyRenderer -testPlatform StandaloneWindows64 -buildTarget StandaloneWindows64 -enabledxrtarget=Oculus -stereorenderingpath=SinglePass -colorspace=Linear -executeMethod Setup.SetupAll -testResults TestResults.xml -logFile UnityLog.txt

You can also open Unity using the cmd line but not include the runTests and automated options. Then run the tests using the Unity Test Runner window. Run all in player is the preferred method. You can run them in the editor but there are currently not any reference images for that configuration. However the failed images won't copy to your results location until you quit the editor (this will be changed in the future) and the test results file doesn't seem to generate when running in this mode but results sometimes show up in the test runner results window (although this is unreliable and the tests listed sometimes can get out of sync with the test case list since the tests are generated by the graphics setup).

Some projects have the setup method exposed in the "Tests" custom menu item which will generate the test case list according to your build settings and rerun light mapping.

Creating new tests

To create a new test in an existing test project:

1. Create a new scene and set up what you want to test.
2. Add the GraphicsTestSettings component to a game object, creating an empty object for this is recommended to be able to find it easily.
3. Set the image comparison settings to what you like. This component will also disable vr tracking to make sure the camera doesn't move.
4. Add a PlatformConfigTestFilters component if you'd like to filter out the case from certain configurations. This is optional and doesn't have to be included if it should run on everything. This is an exclusive list so any config you add won't run the test. You can use the null or none option to include all the configs in a group (such as not to run it on any android config leave all the settings to null or none and set the platform to Android).
5. Add the scene to the build settings and enable it. The setup script will not enable disabled scenes so it is recommended you enable all scenes and let the filters turn them off unless you want to run only one
6. Run the case and it will fail since there is no reference image. Make sure to run SetupGraphicsTestCases.Setup to add the scene to the scene list. The reference image should be generated in your Application.PersistentData directory or copied to the test results location when the editor quits.
7. Copy the reference image to the appropriate Assets/ReferenceImages directory. SetupGraphicsTestCases.Setup will need to be called again before the test run to add the image file to the reference image asset bundle. Now when run the test should pass.

Common Issues

• All screenshots are black - usually because the HMD went to sleep.
• You can turn off the sleep in OpenVR using SteamVR → hamburger menu → Settings → Developer → uncheck "Pause rendering when headset is idle"
• Oculus you can set two registry keys to always have it on. In HKEY_CURRENT_USER\Software\Oculus VR, LLC\Oculus add a AllowScreenBurnIn=1 (Binary) and ProximityState=1 (DWORD)
• The reference image was not found - if the SetupGraphicsTestCases.Setup method has not been run the image won't be in the build and can fail. Or you are running in a configuration for which they need to be generated. Remember many settings can be set by the configuration manager script and so settings may be different from what you think and so can be looking for reference images that aren't there.
• The reference image asset bundle is not found. There were no reference images found for the configuration. Sometimes this can happen if the XR settings fall back to or are set to disabled, usually due to not running the configuration manager script or having that run but setting the xr settings to none by default.
• VR is turned on but the images are monoscopic - usually due to the xr package not being installed correctly. You can remove all xr platforms from the settings and readd them and the package should refresh.