performance/docs/sdk-scenarios.md

SDK Scenarios

An introduction of how to run scenario tests can be found in Scenarios Tests Guide. The current document has specific instruction to run:

• SDK Build Throughput Scenario

SDK Build Throughput Scenario

SDK Build Throughput is a scenario test that measures the throughput of SDK build process. To be more specific, our test implicitly calls

dotnet build <project>

with other applicable arguments and measures its throughput.

There are 2 types of SDK build --- Clean Build and Build No Change.

• Clean Build: simulates the first-time-ever build of the project, when binaries do not exist in the output folder. Between each iteration, the test harness cleans the output folder and turns off dotnet servers.
• Build No Change simulates the build after the first-time-ever build. The test harness runs a warmup build which leaves the binaries, without cleanup between iterations.

Prerequisites

• make sure the test directory is clean of artifacts (you can run post.py to remove existing artifacts folders from the last run )
• python3 or newer
• dotnet runtime 3.1 or newer
• terminal/command prompt in Admin Mode (for collecting kernel traces)
• clean state of the test machine (anti-virus scan is off and no other user program's running -- to minimize the influence of environment on the test)

We will walk through SDK Console Template as an example.

Step 1 Initialize Environment

Same instruction of Scenario Tests Guide - Step 1.

Step 2 Run Precommand

If you are running the test NOT for the first time and app\ folder exists under the asset directory, make sure it's removed so the previous test artifact won't be used. You can use post.py to clean it up:

python3 post.py

Run precommand to create a new console template.

cd emptyconsoletemplate
python3 pre.py default -f net9.0

The default command prepares the asset (creating a new project if the asset is a template and copying it to app\). Now there should be source code of a console template project under app\.

Note that it is important to be aware SDK takes different paths for different TFMs, and you can configure which TFM your SDK tests against. Howeverm your SDK version should be >= the TFM version because SDK cannot build a project that has a newer runtime. Here's a matrix of valid SDK vs. TFM combinations:

	netcoreapp2.1	netcoreapp3.1	net5.0	net6.0	net7.0	net8.0	net9.0
.NET 2.1 SDK	x
.NET 3.1 SDK	x	x
.NET 5 SDK	x	x	x
.NET 6 SDK	x	x	x	x
.NET 7 SDK	x	x	x	x	x
.NET 8 SDK	x	x	x	x	x	x
.NET 9 SDK	x	x	x	x	x	x	x

You can change TFM of the project by specifying -f <tfm>, which allows to replace the <TargetFramework></TargetFramework> property in the project file to be the custom TFM value (make sure it's a valid TFM value) you specified.

Step 3 Run Testcommand

Run testcommand to measure the throughput of sdk build.

For Clean Build test, run:

python3 test.py sdk clean_build

For Build No Change test, run:

python3 test.py sdk build_no_change

The test result should look like the following:

[2020/09/27 23:51:22][INFO] Merging traces\emptycsconsoletemplate_SDK_build_no_change_startup.perflabkernel.etl...
[2020/09/27 23:51:22][INFO] Trace Saved to traces\emptycsconsoletemplate_SDK_build_no_change_startup.etl
[2020/09/27 23:51:22][INFO] Parsing traces\emptycsconsoletemplate_SDK_build_no_change_startup.etl
[2020/09/27 23:51:23][INFO]
[2020/09/27 23:51:23][INFO] Metric         |Average        |Min            |Max
[2020/09/27 23:51:23][INFO] ---------------|---------------|---------------|---------------
[2020/09/27 23:51:23][INFO] Process Time   |1621.667 ms    |1587.230 ms    |1706.269 ms
[2020/09/27 23:51:23][INFO] Time on Thread |969.075 ms     |414.988 ms     |1347.836 ms

Step 4 Run Postcommand

Same instruction of Step 4 in Scenario Tests Guide.

Command Matrix

• <tfm> values:
  • netcoreapp2.1
  • netcoreapp3.1
  • net5.0
  • net6.0
  • net7.0
  • net8.0
  • net9.0
• <build option> values:
  • clean_build
  • build_no_change

Scenario	Asset Directory	Precommand	Testcommand	Postcommand	Supported Framework	Supported Platform
SDK Console Template	emptyconsoletemplate	pre.py default -f <tfm>	test.py sdk <build option>	post.py	netcoreapp2.1;netcoreapp3.1;net5.0;net6.0;net7.0;net8.0;net9.0	Windows;Linux
SDK .NET 2.0 Library Template	netstandard2.0	pre.py default -f <tfm>	test.py sdk <build option>	post.py	netcoreapp2.1;netcoreapp3.1;net5.0;net6.0;net7.0;net8.0;net9.0	Windows;Linux
SDK ASP.NET MVC App Template	mvcapptemplate	pre.py default -f <tfm>	test.py sdk <build option>	post.py	netcoreapp3.1;net5.0;net6.0;net7.0;net8.0;net9.0	Windows;Linux
SDK Web Large 3.0	weblarge3.0	pre.py default -f <tfm>	test.py sdk <build option>	post.py	netcoreapp3.1	Windows;Linux
SDK Windows Forms Large	windowsformslarge	pre.py default -f <tfm>	test.py sdk <build option>	post.py	netcoreapp3.1	Windows
SDK WPF Large	wpflarge	pre.py default -f <tfm>	test.py sdk <build option>	post.py	netcoreapp3.1	Windows
SDK Windows Forms Template	windowsforms	pre.py default -f <tfm>	test.py sdk <build option>	post.py	netcoreapp3.1	Windows
SDK WPF Template	wpf	pre.py default -f <tfm>	test.py sdk <build option>	post.py	netcoreapp3.1	Windows
SDK New Console	emptyconsoletemplate	N/A	test.py sdk new_console	post.py	netcoreapp2.1;netcoreapp3.1;net5.0;net6.0;net7.0;net8.0;net9.0	Windows;Linux