component-detection/docs/running-verification-tests.md

Running Verification Tests On Your Local Machine

Verification tests are used to confirm that no detectors are lost when changes are made to the project. The tests are run on every PR build. They work by comparing the detection results from the main branch to detection results from the new changes on your PR. You can follow the steps below to run them locally.

An automation script is created to setup the test artifact and execute the verification test. Ensure that your local changes builds and execute following powershell script from repository root.

.\test\Microsoft.ComponentDetection.VerificationTests\resources\VerificationTest.ps1

Debug using visual studio

The automation script above will output variables that needs to be replaced in ComponentDetectionIntegrationTests.cs file. Then, verification test can be debugged through Test Explorer by opening \test\Microsoft.ComponentDetection.VerificationTests\Microsoft.DependencyDetective.VerificationTests.csproj in visual studio.

Manual setup

Step 1 : Run Detection on the main branch

• Checkout the main branch in your local repo
• Create a folder to store detection results (e.g C:\old-output-folder)
• Use command line to run detection on the main branch with the code below:

dotnet run scan --Verbosity Verbose --SourceDirectory {path to your local repo} --Output {path to the output folder you created}

For Example:

dotnet run scan --Verbosity Verbose --SourceDirectory C:\componentdetection --Output C:\old-output-folder

Step 2 : Run Detection on your new branch

• Checkout the branch with the new changes you are trying to merge
• Create a folder to store detection results. This folder should be seperate from the one you used in Step 1 (e.g C:\new-output-folder)
• Use command line to run detection on the main branch with the code below:

dotnet run scan --Verbosity Verbose --SourceDirectory {path to your local repo} --Output {path to the output folder you created}

For Example:

dotnet run scan --Verbosity Verbose --SourceDirectory C:\componentdetection --Output C:\new-output-folder

Step 3 : Update variables in the test

• Open the Microsoft.ComponentDetection.VerificationTests project in VS Studio
• Navigate to GatherResources() in ComponentDetectionIntegrationTests.cs
• Update the following variables:
  • oldGithubArtifactsDir : This should be the output folder you created in Step 1
  • newGithubArtifactsDir : This should be the output folder you created in Step 2
  • allowedTimeDriftRatioString: This should be ".75"

Step 4: Run The tests

You can run the tests in two ways:

• Run or Debug the tests in test explorer.
• Use the command line to navigate to the Microsoft.ComponentDetection.VerificationTests folder, and run dotnet test.