msquic/docs/PGO.md

Profile Guided Optimizations

MsQuic uses profile-guided optimizations (PGO) to generate optimized builds of the MsQuic library. PGO lets you optimize the whole library by using data from a previous run of the library.

Note This document is Windows specific.

Build

During the build for x86 and x64 release builds (arm/arm64 are currently unsupported) a profile-guided database file (.pgd), generated from a previous run, is passed to the linker. The linker uses this data to optimize the new build.

Build for Training

> ./scripts/build.ps1 -Config Release -PGO

By default, the library is not built in "training mode". To enable this, you must pass the -PGO switch to the build.ps1 PowerShell script. This configures the linker to configure the library so that it can be trained. Whenever the library unloads a .pgc file will be dumped to the local directory. This file can be used update the existing .pgd file.

Training

A fundamental part of profile-guided optimizations is training. The code is run through production scenarios while in "training mode" to generate a data set that can be used for a future build to optimize for the scenario.

1. Build for training.
2. Copy the binaries to the test machine(s).
   1. The PGO msquic library.
   2. The test tool (e.g. secnetperf).
   3. The PGO runtime library from your VS install: (e.g. "C:\Program Files (x86)\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.26.28801\bin\Hostx64\x64\pgort140.dll").
3. Run the test for the production/performance scenario.
4. Use pgomgr to merge the .pgc into the .pgd.
5. Update the .pgd and .pdb files in the repository.

Kernel mode Profile Guided Optimizations

PGO for kernel mode is a much more manual process than user mode. These steps guide you through the process.

Build

To build for kernel PGO, copy pgortsys.lib locally and edit the msquicpriv.kernel.vcxproj file with the following patch:

diff --git a/src/bin/winkernel/msquicpriv.kernel.vcxproj b/src/bin/winkernel/msquicpriv.kernel.vcxproj
index 04d89207..b125cdd7 100644
--- a/src/bin/winkernel/msquicpriv.kernel.vcxproj
+++ b/src/bin/winkernel/msquicpriv.kernel.vcxproj
@@ -78,7 +78,7 @@
   <ImportGroup Label="PropertySheets">
     <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
   </ImportGroup>
-  <PropertyGroup Label="UserMacros" />
+  <PropertyGroup><LibraryPath>$(LibraryPath);$(VC_LibraryPath_VC_x64_Desktop);c:\path\to\directory\containing\pgortsys.lib\</LibraryPath></PropertyGroup>
   <PropertyGroup>
     <QUIC_VER_BUILD_ID Condition="'$(QUIC_VER_BUILD_ID)' == ''">0</QUIC_VER_BUILD_ID>
     <QUIC_VER_SUFFIX Condition="'$(QUIC_VER_SUFFIX)' == ''">-private</QUIC_VER_SUFFIX>
@@ -106,7 +106,7 @@
     </ClCompile>
     <Link>
       <ModuleDefinitionFile>msquicpriv.src</ModuleDefinitionFile>
-      <AdditionalDependencies>cng.lib;ksecdd.lib;msnetioid.lib;ndis.lib;netio.lib;uuid.lib;%(AdditionalDependencies)</AdditionalDependencies>
+      <AdditionalDependencies>cng.lib;ksecdd.lib;msnetioid.lib;ndis.lib;netio.lib;uuid.lib;pgortsys.lib;%(AdditionalDependencies)</AdditionalDependencies>
       <AdditionalOptions>/kernel /NOOPTIDATA /pdbcompress /MERGE:.gfids=GFIDS /MERGE:.orpc=.text /MERGE:_PAGE=PAGE /MERGE:_RDATA=.rdata /MERGE:_TEXT=.text /section:GFIDS,d</AdditionalOptions>
       <LinkTimeCodeGeneration>UseLinkTimeCodeGeneration</LinkTimeCodeGeneration>
     </Link>
@@ -126,7 +126,7 @@
   </ItemDefinitionGroup>
   <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
     <Link>
-      <AdditionalOptions>%(AdditionalOptions) /FORCE:PGOREPRO /USEPROFILE:PGD=$(SolutionDir)artifacts\bin\winkernel\$(Platform)_$(Configuration)_schannel\priv\msquic.pgd</AdditionalOptions>
+      <AdditionalOptions>%(AdditionalOptions) /FORCE:PGOREPRO /LTCG:PGI /Profile</AdditionalOptions>
     </Link>
   </ItemDefinitionGroup>
   <ItemGroup>

You should clean build for x64 and Release.

Then build user mode MsQuic using the regular script with the following flags -PGO -Tls schannel -config Release.

Setting up the perf machines

To train kernel mode PGO, we use the secnetperf utility. To configure the perf machines, run the following powershell script as Administrator:

param(
    # Build path
    [Parameter(Mandatory=$true)]
    [string]$BuildPath,

    # MsQuic location
    [Parameter(Mandatory=$true)]
    [string]$MsQuicPath,

    # CPU Architecture
    [Parameter(Mandatory=$false)]
    [ValidateSet("amd64", "x86", "arm", "arm64")]
    [string]$Arch="amd64"
)

$PgoBasePath = "$($BuildPath)\$($Arch)fre\pgo"

if (!(Test-Path -Path "c:\pgo_collection")) {
    New-Item -Path "c:\pgo_collection" -ItemType Directory -Force | Out-Null
}

Copy-Item -Path "$($PgoBasePath)\pgoapi\*" -Filter *.ps1 -Destination "C:\pgo_collection"
Copy-Item -Path "$($BuildPath)\$($Arch)fre\bin\idw\signTool.exe" -Destination "C:\pgo_collection"
Copy-Item -Path "$($BuildPath)\$($Arch)fre\bin\idw\signTool.dll" -Destination "C:\pgo_collection"

if (!(Test-Path -Path "c:\pgo_collection\sweep")) {
    New-Item -Path "c:\pgo_collection\sweep" -ItemType Directory -Force | Out-Null
}

Copy-Item -Path "$($PgoBasePath)\kmode\system32\pgosweep.exe"          -Destination "C:\pgo_collection\sweep"
Copy-Item -Path "$($PgoBasePath)\kmode\system32\vcruntime140.dll"      -Destination "C:\pgo_collection\sweep"
Copy-Item -Path "$($PgoBasePath)\kmode\system32\drivers\pgodriver.sys" -Destination "C:\pgo_collection\sweep"
Copy-Item -Path "$($PgoBasePath)\bbttools\bbtlddll.exe"                -Destination "C:\pgo_collection\sweep"
Copy-Item -Path "$($PgoBasePath)\bbttools\createdir.sys"               -Destination "C:\pgo_collection\sweep"
Copy-Item -Path "$($PgoBasePath)\tools\pgort140.dll"                   -Destination "C:\pgo_collection\sweep"

# Install PGO
powershell -NonInteractive -NoProfile -File "c:\pgo_collection\install-pgo.ps1" -PgoChunkPath $PgoBasePath -PgoChunkNativePath $PgoBasePath -PgoCollectionDir "c:\pgo_collection" -SignTool "c:\pgo_collection\signtool.exe" -PGOArch $Arch

# Fix PGODriver
sc.exe config pgodriver start= auto

# Copy MsQuic
if ($arch -eq "amd64") {
    $QuicArch = "x64"
} else {
    $QuicArch = $Arch
}

$WindowsBinPath = "artifacts\bin\windows\$($QuicArch)_Release_schannel"
$WinKernelBinPath = "artifacts\bin\winKernel\$($QuicArch)_Release_schannel"

if (!(Test-Path -Path "c:\msquic\$($WindowsBinPath)")) {
    New-Item -Path "c:\msquic\$($WindowsBinPath)" -ItemType Directory -Force | Out-Null
}

if (!(Test-Path -Path "c:\msquic\$($WinKernelBinPath)")) {
    New-Item -Path "c:\msquic\$($WinKernelBinPath)" -ItemType Directory -Force | Out-Null
}

Copy-Item -Path "$($MsQuicPath)\$($WindowsBinPath)\msquic.dll"   -Destination "c:\msquic\$($WinKernelBinPath)"
Copy-Item -Path "$($MsQuicPath)\$($WindowsBinPath)\secnetperf.exe" -Destination "c:\msquic\$($WinKernelBinPath)"
Copy-Item -Path "$($MsQuicPath)\$($WinKernelBinPath)\*"          -Destination "c:\msquic\$($WinKernelBinPath)"

sc.exe create "msquicpriv" type= kernel binpath= "C:\msquic\$($WinKernelBinPath)\msquicpriv.sys" start= demand

New-NetFirewallRule -DisplayName "Allow SecNetPerf" -Enabled True -Profile Any -Direction Inbound -Action Allow -Protocol UDP -LocalPort 4433

bcdedit /debug on

Write-Host Now Reboot the machine

Make sure the machine is configured for kernel debugging and a kernel debugger is attached, otherwise the msquicpriv.sys driver won't load.

Training

Now that the perf machines are configured for kernel mode PGO, it's time to run the scenarios.

On the machine that will act as server, run the following command to start the server:

secnetperf.exe -kernel

Once running, clear the PGO counts on both the client and server machines to get a clean slate:

pgosweep.exe /driver msquicpriv.sys .\msquicpriv.pgc
del msquicpriv.pgc

On the machine acting as client, run the following commands to generate traffic:

secnetperf.exe --kernel -test:tput -target:<server IP> -upload:5000000000
secnetperf.exe --kernel -test:tput -target:<server IP> -download:5000000000
secnetperf.exe --kernel -test:RPS -target:<server IP>
secnetperf.exe --kernel -test:HPS -target:<server IP>

After the client finishes all scenarios, run this again on the client and the server to collect the updated counts:

pgosweep.exe /driver msquicpriv.sys .\msquicpriv.pgc

Copy the .pgc files to a machine with the Visual Studio tools installed and run the following to merge the PGC files into the PGD that was generated during build, and then copy that PGD into the git repo:

pgomgr.exe /merge msquicpriv-client.pgc msquicpriv-server.pgc c:\msquic\artifacts\bin\winkernel\x64_Release_schannel\msquicpriv.pgd
xcopy :\msquic\artifacts\bin\winkernel\x64_Release_schannel\msquicpriv.pgd c:\msquic\src\bin\winkernel\pgo_x64\msquic.pgd

If pgomgr.exe emits a warning that the database doesn't match, use pgodump.exe to check the ID of both the PGD and PGC files and ensure they match. If the PGC file doesn't match, it's most likely because msquicpriv.sys didn't unload completely when you installed a new one. You will need to stop PGODriver.sys before stopping msquicpriv.sys when replacing msquicpriv.sys with a newer version. You can also reboot the system after replacing msquicpriv.sys.

Sample output from pgodump.exe showing the IDs are same between PGD and PGC

> pgodump.exe c:\msquic\artifacts\bin\winkernel\x64_Release_schannel\msquicpriv.pgd
Microsoft (R) Profile Guided Optimization Database Dump Utility
Copyright (C) Microsoft Corporation. All rights reserved.

PGD File: c:\msquic\artifacts\bin\winkernel\x64_Release_schannel\msquicpriv.pgd (v44, ID CC21BC03, Signature 4F474F50)  10/27/2020 22:25:53

> pgodump.exe msquicpriv-client.pgc
Microsoft (R) Profile Guided Optimization Database Dump Utility
Copyright (C) Microsoft Corporation. All rights reserved.

PGC File: msquicpriv!client.pgc (ID CC21BC03)