Benchmarks/scenarios

Benchmark Scenarios

This folder contains benchmarks that represent common scenarios to look at for the .NET team.

Continuous benchmarking results are available on this PowerBI dashboard.

Note: The dashboard's navigation is at the bottom of the page, accessible by clicking between the Next/Previous page links.

Requirements

These jobs can be executed using the .NET Crank global tool. .NET Core is required to install the global tool.

Install crank with the following command:

dotnet tool install Microsoft.Crank.Controller --version "0.2.0-*" --global

Alternatively, update crank with the following command:

dotnet tool update Microsoft.Crank.Controller --version "0.2.0-*" --global

Profiles

Each profile defines a set of machines, private IPs and ports that are used to run a benchmark.

Profile	Arch	OS	Proc
local	(local machine)	(local machine)	(local machine)
aspnet-perf-lin	INTEL, 12 logical cores, 1 socket, 32GB	Ubuntu 22.04, Kernel 5.15.0	Intel(R) Xeon(R) E-2336 CPU @ 2.90GHz
aspnet-perf-win	INTEL, 12 logical cores, 1 socket, 32GB	Windows Server 2022	Intel(R) Xeon(R) E-2336 CPU @ 2.90GHz
aspnet-citrine-lin	INTEL, logical 28 cores, 1 socket, 32GB	Ubuntu 20.04, Kernel 5.4.0	Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz
aspnet-citrine-win	INTEL, logical 28 cores, 1 socket, 32GB	Windows Server 2022	Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz
aspnet-citrine-amd	AMD, logical 48 cores, 1 socket, 64GB, 1 NUMA	Ubuntu 20.04, Kernel 5.4.0	AMD EPYC 7402P 24-Core Processor
aspnet-citrine-amd2	AMD, logical 48 cores, 1 socket, 64GB, 4 NUMA	Ubuntu 20.04, Kernel 5.4.0	AMD EPYC 7402P 24-Core Processor
aspnet-citrine-arm-lin	ARM64, 80 logical cores, 1 socket, 1 NUMA, 128 GB	Ubuntu 20.04, Kernel 5.4.0	Neoverse-N1
aspnet-gold-lin	INTEL, 56 logical cores, 1 socket, 1 NUMA, 64 GB	Ubuntu 22.04, Kernel 6.8.0-45-generic	Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz
aspnet-gold-win	INTEL, 56 logical cores, 1 socket, 1 NUMA, 64 GB	Windows Server 2022	Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz
aspnet-siryn-arm-lin	ARM64, 128 logical cores, 1 socket, 1 NUMA, 250GB	Ubuntu 20.04, Kernel 5.4.0	Ampere(R) AmpereOne(TM)

For testing purpose only, the local profile requires a local agent to run:

dotnet tool install Microsoft.Crank.Agent --version "0.2.0-*" --global

crank-agent

Plaintext benchmarks

The source code for these benchmarks is located here. These scenarios return a "Hello World" string and the client uses HTTP pipelining with 16 requests.

Sample

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/plaintext.benchmarks.yml --scenario plaintext --profile aspnet-perf-lin

Available scenarios

• plaintext: Middleware implementation
• https: Middleware implementation, using HTTPS
• endpoint: Middleware implementation with Endpoint routing
• mvc: Controller implementation
• mapaction: Endpoint routing implementation
• connectionclose: Middleware implementation, the connection is closed after each request. Pipelining is disabled.
• connectionclosehttps: Same as connectionclose but over https

Json benchmarks

The source code for these benchmarks is located here. These scenarios serialize and return a { "message": "Hello World" } string.

The serialization is done with System.Text.Json.

Sample

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/json.benchmarks.yml --scenario json --profile aspnet-perf-lin

Available scenarios

• json: Middleware implementation
• https: Middleware implementation, using HTTPS
• mvc: Controller implementation
• mapaction: Endpoint routing implementation

Database benchmarks

The source code for these benchmarks is located here. These scenarios execute some database requests and return either HTML or Json.

The database server is PostgresQL.

Sample

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/database.benchmarks.yml --scenario fortunes --profile aspnet-perf-lin

Available scenarios

The following scenarios are implemented from a middleware (no MVC)

• fortunes
• fortunes_ef
• fortunes_dapper
• single_query
• single_query_ef
• single_query_dapper
• multiple_queries
• multiple_queries_ef
• multiple_queries_dapper
• updates
• updates_ef
• updates_dapper

The following scenarios are using ASP.NET CORE MVC

• fortunes_ef_mvc_https

The suffixes represent different database access strategies:

• No suffix: Raw ADO.NET
• "ef" suffix: Entity Framework Core
• "dapper" suffix: Dapper

Platform benchmarks

The source code for these benchmarks is located here. These scenarios are highly optimized to provide the best performance, in detriment of extensibility and code complexity.

The database server is PostgresQL.

Sample

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/platform.benchmarks.yml --scenario fortunes --profile aspnet-perf-lin

Available scenarios

• plaintext
• json
• fortunes
• fortunes_ef
• fortunes_dapper
• single_query
• multiple_queries
• updates
• caching

Proxy benchmarks

These scenarios are running several web proxies, including YARP.

The downstream service returns a variable size content. By default the result is 10 bytes.

Sample

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/proxy.benchmarks.yml --scenario proxy-httpclient --profile aspnet-perf-lin

Available scenarios

• proxy-yarp
• proxy-httpclient
• proxy-nginx
• proxy-haproxy
• proxy-envoy

Custom arguments

Response size

The size of the payload can be changed by adapting the path of the requested url:

--variable path=/?s=100

Protocols

The server and downstream protocols can be changed to http (default), https and h2. The following example shows how to use "h2 - https":

--variable serverScheme=https --variable serverProtocol=http2 --variable downstreamScheme=https --variable downstreamProtocol=https

NOTES: nginx doesn't support http2 on the upstream server c.f. https://www.nginx.com/blog/http2-module-nginx/#QandA

Body size

Custom bodies can be used with the bodyFile and verb variables like this:

--variable bodyFile=https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/assets/100B.txt --variable verb=POST

Local and remote files can be used.

Headers

As for many other scenarios, the existing web load clients (wrk, bombardier, ...) are configured to support predefined headers:

• none (default)
• plaintext: "Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7"
• json: "Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7"
• html: "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"

The variable presetHeaders is used to select one of these:

--variable presetHeaders=plaintext

Frameworks benchmarks

These scenarios measure the performance of different other frameworks

• NodeJs (JavaScript)
• Bun (TypeScript)
• Actix (Rust)
• Xitca (Rust)
• FastHttp (Go)
• Netty (Java)
• Vertx (Java)
• Wizzardo (Java)
• Spring (Java)
• Gin (Go)
• Express (JavaScript)

Sample

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/te.benchmarks.yml --scenario plaintext_nodejs --profile aspnet-perf-lin

Available scenarios

• plaintext_aspnetcore, json_aspnetcore, fortunes_aspnetcore
• plaintext_nodejs, json_nodejs, fortunes_nodejs
• plaintext_bun, json_bun
• plaintext_actix, json_actix, fortunes_actix
• plaintext_xitca, json_xitca, fortunes_xitca
• plaintext_fasthttp, json_fasthttp, fortunes_fasthttp
• plaintext_netty, json_netty
• plaintext_wizzardo, json_wizzardo, single_query_wizzardo, multiple_queries_wizzardo, updates_wizzardo, cached_queries_wizzardo
• plaintext_vertx, json_vertx, single_query_vertx, multiple_queries_vertx, updates_vertx, cached_queries_vertx
• plaintext_spring, json_spring, single_query_spring, multiple_queries_spring, updates_spring
• plaintext_gin, json_gin, fortunes_gin
• plaintext_express, json_express, fortunes_express

Grpc benchmarks

These scenarios measure the performance of different Grpc server and clients implementations.

• Go
• Native (C)
• ASP.NET

Sample

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/grpc.benchmarks.yml --scenario grpcaspnetcoreserver-grpcnetclient --profile aspnet-perf-lin --variable streams=70 --variable connections=1

Available scenarios

• grpcaspnetcoreserver-grpcnetclient
• grpccoreserver-grpcnetclient
• grpcgoserver-grpcnetclient
• grpcaspnetcoreserver-grpccoreclient
• grpccoreserver-grpccoreclient
• grpcgoserver-grpccoreclient
• grpcaspnetcoreserver-grpcgoclient
• grpccoreserver-grpcgoclient
• grpcgoserver-grpcgoclient
• grpcaspnetcoreserver-h2loadclient
• grpccoreserver-h2loadclient
• grpcgoserver-h2loadclient

Arguments

• Number of streams:
  • --variable streams=1
  • --variable streams=70
• Number of connections:
  • --variable connections=1
  • --variable connections=28
• Protocol:
  • --variable protocol=h2
  • --variable protocol=h3
  • --variable protocol=h2c
• Call types:
  • Unary: --variable scenario=unary
  • Server streaming: --variable scenario=serverstreaming
  • Ping ping streaming: --variable scenario=pingpongstreaming

Static file benchmarks

Middleware based application that serve static files of any size.

Sample

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/staticfiles.benchmarks.yml --scenario static --profile aspnet-perf-lin

Available scenarios

• static

The filename and size can be changed by adapting these variables:

--variable sizeInBytes=1024
--variable filename=file.txt

This scenario can easily reach the max network bandwidth. To verify that use --load.options.displayOutput true which will display the wrk output, including the transfer rate. Example of saturated network for a 40Gb/s NIC (Citrine).

[load] Transfer/sec:      4.37GB

SignalR benchmarks

These scenarios are running various SignalR benchmarks. The transport and serialization methods can be configured.

Sample

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/signalr.benchmarks.yml --scenario signalr --profile aspnet-perf-lin --variable scenario=echo --variable transport=websockets --variable protocol=messagepack

Available scenarios

• signalr

Arguments

• Scenario:
  • --variable scenario=broadcast
  • --variable scenario=echo
  • --variable scenario=echoAll
• Transport:
  • --variable transport=websockets
  • --variable transport=serversentevents
  • --variable transport=longpolling
• Protocol:
  • --variable protocol=json
  • --variable protocol=messagepack

Note: MessagePack is not supported with ServerSentEvents

WebSockets benchmarks

These scenarios are running various WebSockets benchmarks.

Sample

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/websocket.benchmarks.yml --scenario websocket --profile aspnet-perf-lin --variable scenario=echo

Available scenarios

• websocket

Arguments

• Scenario:
  • --variable scenario=echo

Orchard benchmarks

These scenarios are running various Orchard Core CMS benchmarks with either Sqlite or PostgresQL.

Sample

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/orchard.benchmarks.yml --scenario about-sqlite --profile aspnet-perf-win

Available scenarios

• about-sqlite: Simple about page using Sqlite
• about-postgresql: Simple about page using PostgresQL

Distributed cache benchmarks

These scenarios are running various distributed cache benchmarks.

For all the scenarios, the store is initialized with CacheCount cache entries. Each request will issue a read or a write based on the WriteRatio argument choosing a key randomly. The HTTP response won't contain the cache entry data so that it doesn't impact the raw store perf measurement.

Sample

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/redis.benchmarks.yml --scenario redis --profile aspnet-perf-lin

Available scenarios

• redis: Read/Write cache entries using StackExchangeRedisCache distributed cache provider.
• redis_local: same as redis, with the Redis server on the same machine as the application.
• memory: Read/Write cache entries using MemoryCache distributed cache provider.
• nullcache: Uses a fake implementation of IDistributedCache that does nothing when invoked.

Arguments

• cacheCount (256): The number or cache entries created in the store.
  • --variable cacheCount=256
• keyLength (16): The size or the keys. They are generated using random alphanumeric characters.
  • --variable keyLength=16
• contentLength (64): The size of the cache entries.
  • --variable contentLength=64
• writeRatio (0): The ratio between reads and writes. Set to 100 for only writes.
  • --variable writeRatio=0

Native AOT benchmarks

These scenarios provide benchmarks to help improve the Native AOT performance.

Sample

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/goldilocks.benchmarks.yml --scenario basicminimalapipublishaot --profile aspnet-perf-lin

Available scenarios

• basicminimalapivanilla
• basicminimalapipublishaot
• basicgrpcvanilla
• basicgrpcpublishaot

HTTP client benchmarks

These scenarios are running various HTTP client benchmarks.

Sample

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/httpclient.benchmarks.yml --scenario httpclient-kestrel-get --profile aspnet-perf-lin --variable concurrencyPerHttpClient=200

Available scenarios

• httpclient-kestrel-get: Execute GET requests to the server. Client is HttpClient, server is Kestrel (minimal API).
• httpclient-kestrel-post: Execute POST requests to the server. Client is HttpClient, server is Kestrel (minimal API).
• httpclient-kestrel-post-slow: Execute POST requests to the server, upload content 1 byte at a time. Similar to httpclient-kestrel-post, with pre-defined parameters (requestContentWriteSize=1, requestContentFlushAfterWrite=true, requestContentUnknownLength=true)
• wrk-kestrel-get: Execute GET requests to the server. Client is WRK, server is Kestrel (minimal API).

Arguments

• Server port to use:
  • --variable serverPort=<N> (default: 5000)
• HTTP Version to use:
  • --variable httpVersion="1.1" (default)
  • --variable httpVersion="2.0"
  • --variable httpVersion="3.0" -- requires HTTPS
• Whether to use HTTPS:
  • --variable useHttps=false (default)
  • --variable useHttps=true
• Pre-defined request headers:
  • --variable requestHeaders=none (default)
  • --variable requestHeaders=connectionclose
  • --variable requestHeaders=expectcontinue
• Number of generated request headers with static values:
  • --variable generatedStaticRequestHeadersCount=<N> (default: 0)
• Number of generated request headers with values changing per each request:
  • --variable generatedDynamicRequestHeadersCount=<N> (default: 0)
• Response content size in bytes:
  • --variable responseSize=<N> (default: 0, meaning no content)
• Server's HTTP/3 stream limit:
  • --variable http3StreamLimit=<N> (default: 0, meaning server's default. Max value is 65535)
• Number of HTTP clients (HttpClient in httpClient job, connection in WRK):
  • --variable numberOfHttpClients=<N> (default: 1)
• Number of concurrect requests per one HTTP client -- unsupported by WRK:
  • --variable concurrencyPerHttpClient=<N> (default: 1)
• Whether to collect request timings (time to headers, to first content byte, to last content byte) -- unsupported by WRK:
  • --variable collectRequestTimings=false (default)
  • --variable collectRequestTimings=true
• Duration of the warmup in seconds:
  • --variable warmup=<N> (default: 15)
• Duration of the test in seconds:
  • --variable duration=<N> (default: 15)

POST-specific arguments:

• Request content size in bytes:
  • --variable requestContentSize=<N> (default: 1024 for POST scenarios, 0 for GET scenarios)
• How many bytes (at max) will be written per each write of request content (also chunk size if chunked encoding is used):
  • --variable requestContentWriteSize=<N> (default: 81920)
• Whether to flush request content stream after each write:
  • --variable requestContentFlushAfterWrite=false (default)
  • --variable requestContentFlushAfterWrite=true
• Whether request content length should be unknown to server. If false, will send Content-Length header, if true, will use chunked encoding for HTTP/1.1 or unknown content length for HTTP/2.0 and 3.0:
  • --variable requestContentUnknownLength=false (default)
  • --variable requestContentUnknownLength=true

HttpClient-specific arguments:

• Max number of HTTP/1.1 connections per server:
  • --variable http11MaxConnectionsPerServer=<N> (default: 0, meaning no limit)
• Enable multiple HTTP/2.0 connections:
  • --variable http20EnableMultipleConnections=true (default)
  • --variable http20EnableMultipleConnections=false
• Enable multiple HTTP/3.0 connections (from .NET 9.0):
  • --variable http30EnableMultipleConnections=true (default)
  • --variable http30EnableMultipleConnections=false
• Whether to use WinHttpHandler instead of SocketsHttpHandler:
  • --variable useWinHttpHandler=false (default)
  • --variable useWinHttpHandler=true -- requires Windows
• Whether to use HttpMessageInvoker instead of HttpClient:
  • --variable useHttpMessageInvoker=false (default)
  • --variable useHttpMessageInvoker=true
• Whether to use HttpClient.DefaultRequestHeaders for pre-set and static headers:
  • --variable useDefaultRequestHeaders=false (default)
  • --variable useDefaultRequestHeaders=true

Blazor benchmarks

These scenarios provide benchmarks for Blazor.

Sample

crank --config https://raw.githubusercontent.com/dotnet/performance/main/src/benchmarks/gc/scenarios/CrankConfiguration.yaml --scenario ssr --profile aspnet-perf-lin

GC benchmarks

These scenarios provide benchmarks to help improve the performance of the .NET GC.

Sample

crank --config https://raw.githubusercontent.com/dotnet/performance/main/src/benchmarks/gc/scenarios/CrankConfiguration.yaml --scenario 2gb-pinning --profile aspnet-citrine-win --application.framework net10.0

Available scenarios

• 0gb - Server
• 2gb - Server
• 2gb-pinning - Server
• 20gb - Server
• 0gb - Workstation
• 2gb - Workstation
• 2gb-pinning - Workstation
• 20gb - Workstation

Micro benchmarks

These scenarios are running dotnet micro benchmarks from the https://github.com/dotnet/performance repository.

Sample

crank --config https://github.com/aspnet/benchmarks/blob/main/scenarios/dotnet.benchmarks.yml?raw=true --scenario linq --profile aspnet-perf-win

Available scenarios

• linq
• sockets

The scenario named custom can be used to pass any custom filter variable like so:

crank --config https://github.com/aspnet/benchmarks/blob/main/scenarios/dotnet.benchmarks.yml?raw=true --scenario custom --profile aspnet-perf-win --variable filter=*LinqBenchmarks*

FAQ

The following command lines assume that the job to configure is named application which should be the name used in most of the configuration defined in this document.

How to use the latest .NET version?

By default the pre-configured scenarios are using the latest official release of .NET. The framework argument of a service can change which TFM to target.

Example:

Using the daily builds of .NET by targeting net9.0 for the application service.

--application.framework net10.0

How to benchmark a custom fork and/or branch?

If you can create a local branch to this repository, then the argument to add is:

--application.source.branchOrCommit mybranch

If your branch is in another fork then you also need to change the repository value:

--application.source.respository https://github.com/myorg/myrepo

How to upload custom files?

Local files or complete folders can be sent to the running agents to path runtime and aspnet core files:

--application.options.outputFiles c:\build\System.Private.CoreLib.dll

or for a folder:

--application.options.outputFiles c:\build\kestrel

This argument can be repeated in case there are multiple sources.

Running with specific runtime versions to isolate regressions

The list of public builds for ASP.NET and Core CLR are available on these feeds:

• ASP.NET: https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet9/nuget/v3/flat2/Microsoft.AspNetCore.App.Runtime.linux-x64/index.json
• Core CLR: https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet9/nuget/v3/flat2/Microsoft.NetCore.App.Runtime.linux-x64/index.json

Use --application.runtimeVersion x.y.z and --application.aspnetCoreVersion x.y.z to isolate which build, and ultimately which commit introduced a regression.

Getting the command line from Power BI

Most pages in the Power BI dashboard list the crank command lines that were used. Select a benchmark and an environment to filter the table:

Collecting crash dumps

.NET can collect crash dumps automatically using specific https://learn.microsoft.com/en-us/dotnet/core/diagnostics/collect-dumps-crash

Here is how to do it with crank and download the crash dump automatically.

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/plaintext.benchmarks.yml --scenario plaintext --profile aspnet-perf-lin --application.environmentVariables DOTNET_DbgEnableMiniDump=1 --application.environmentVariables DOTNET_DbgMiniDumpType=4 --application.environmentVariables DOTNET_DbgMiniDumpName=mydump --application.environmentVariables DOTNET_CreateDumpVerboseDiagnostics=1 --application.options.downloadfiles mydump

Note that these ENVs could be set for all runs and crank will run just fine if it can't download the file because no crash happened.

Downloading binlogs

When running a benchmark, the binlog can be produced and downloaded by adding these arguments:

--application.buildArguments "-bl" --application.options.downloadFiles "../msbuild.binlog"