This pipeline does everything the existing release pipeline does, except
it does it using the OneBranch official templates.
Most of our existing build infrastructure has been reused, with the
following changes:
- We are no longer using `job-submit-windows-vpack`, as OneBranch does
this for us.
- `job-merge-msix-into-bundle` now supports afterBuildSteps, which we
use to stage the msixbundle into the right place for the vpack
- `job-build-project` supports deleting all non-signed files (which the
OneBranch post-build validation requires)
- `job-build-project` now deletes `console.dll`, which is unused in any
of our builds, because XFGCheck blows up on it for some reason on x86
- `job-publish-symbols` now supports two different types of PAT
ingestion
- I have pulled out the NuGet filename variables into a shared variables
template
I have also introduced a TSA config (which files bugs on us for binary
analysis failures as well as using the word 'sucks' and stuff.)
I have also baselined a number of control flow guard/binary analysis
failures.
After the nightly build completes, we'll automatically generate a
.appinstaller and publich it plus the msixbundle to an Azure Storage
account.
I had to add step/job customization to the publish step in the full
release pipeline template.
The .appinstaller hardcodes our XAML dependency, which makes it a bit of
a pain. We can revisit this later, and publish our dependencies
directly and automatically instead of hardcoding them.
I am considering moving the appinstaller generation step to the MSIX
bundling job, but this works right now and is not too terrible.
Closes#774
This pull request moves HwndTerminal into Microsoft.Terminal.Control.Lib
and removes PublicTerminalCore completely.
Microsoft.Terminal.Control.dll now exports the C API from HwndTerminal.
This adds ~100kb to Microsoft.Terminal.Control.dll and ~1400kb to the
WPF package (per architecture) but with the coming interactivity
platform merge it's going to benefit us big time.
This pull request rewrites the entire Azure DevOps build system.
The guiding principles behind this rewrite are:
- No pipeline definitions should contain steps (or tasks) directly.
- All jobs should be in template files.
- Any set of steps that is reused across multiple jobs must be in
template files.
- All artifact names can be customized (via a property called
`artifactStem` on all templates that produce or consume artifacts).
- No compilation happens outside of the "Build" phase, to consolidate
the production and indexing of PDBs.
- **Building the project produces a `bin` directory.** That `bin`
directory is therefore the primary currency of the build. Jobs will
either produce or consume `bin` if they want to do anything with the
build outputs.
- All step and job templates are named with `step` or `job` _first_,
which disambiguates them in the templates directory.
- Most jobs can be run on different `pool`s, so that we can put
expensive jobs on expensive build agents and cheap jobs on cheap
build agents. Some jobs handle pool selection on their own, however.
Our original build pipelines used the `VSBuild` task _all over the
place._ This resulted in Terminal being built in myriad ways, different
for every pipeline. There was an attempt at standardization early on,
where `ci.yml` consumed jobs and steps templates... but when
`release.yml` was added, all of that went out the window.
The new pipelines are consistent and focus on a small, well-defined set
of jobs:
- `job-build-project`
- This is the big one!
- Takes a list of build configurations and platforms.
- Produces an artifact named `build-PLATFORM-CONFIG` for the entire
matrix of possibilities.
- Optionally signs the output and produces a bill of materials.
- Admittedly has a lot going on.
- `job-build-package-wpf`
- Takes a list of build configurations and platforms.
- Consumes the `build-` artifact for every config/platform
possibility, plus one for "Any CPU" (hardcoded; this is where the
.NET code builds)
- Produces one `wpf-nupkg-CONFIG` for each configuration, merging
all platforms.
- Optionally signs the output and produces a bill of materials.
- `job-merge-msix-into-bundle`
- Takes a list of build configurations and platforms.
- Consumes the `build-` artifact for every config/platform
- Produces one `appxbundle-CONFIG` for each configuration, merging
all platforms for that config into one `msixbundle`.
- Optionally signs the output and produces a bill of materials.
- `job-package-conpty`
- Takes a list of build configurations and platforms.
- Consumes the `build-` artifact for every config/platform
- Produces one `conpty-nupkg-CONFIG` for each configuration, merging
all platforms.
- Optionally signs the output and produces a bill of materials.
- `job-test-project`
- Takes **one** build config and **one** platform.
- Consumes `build-PLATFORM-CONFIG`
- Selects its own pools (hardcoded) because it knows about
architectures and must choose the right agent arch.
- Runs tests (directly on the build agent).
- `job-run-pgo-tests`
- Just like the above, but runs tests where `IsPgo` is `true`
- Collects all of the PGO counts and publishes a `pgc-intermediates`
artifact for that platform and configuration.
- `job-pgo-merge-pgd`
- Takes **one** build config and multiple platforms.
- Consumes `build-$platform-CONFIG` for each platform.
- Consumes `pgc-intermediates-$platform-CONFIG` for each platform.
- Merges the `pgc` files into `pgd` files
- Produces a new `pgd-` artifact.
- `job-pgo-build-nuget-and-publish`
- Consumes the `pgd-` artifact from above.
- Packs it into a `nupkg` and publishes it.
- `job-submit-windows-vpack`
- Only expected to run against `Release`.
- Consumes the `appxbundle-CONFIG` artifact.
- Publishes it to a vpack for Windows to consume.
- `job-check-code-format`
- Does not use artifacts. Runs `clang-format`.
- `job-index-github-codenav`
- Does not use artifacts.
Fuzz submission is broken due to changes in the `onefuzz` client.
I have removed the compliance and security build because it is no longer
supported.
Finally, this pull request has some additional benefits:
- I've expanded the PGO build phase to cover ARM64!
- We can remove everything Helix-related except the WTT parser
- We no longer depend on Helix submission or Helix pools
- The WPF control's inner DLLs are now codesigned (#15404)
- Symbols for the WPF control, both .NET and C++, are published
alongside all other symbols.
- The files we submit to ESRP for signing are batched up into a single
step[^1]
Closes#11874Closes#11974Closes#15404
[^1]: This will have to change if we want to sign the individual
per-architecture `.appx` files before bundling so that they can be
directly installed.
This pull request removes, in full, our dependency on cpprestsdk. This
allows us to shed 500KiB-1.2MiB from our package off the top and enables
the following future investments:
- Removal of the App CRT forwarders to save an additional ~500KiB
- Switching over to the HybridCRT and removing our dependency on _any
CRT_.
cpprest was built on my dev box two or so years ago, and is in _utter_
violation of our compliance guidelines on SBOM et al.
In addition, this change allows us to use the proxy server configured
in Windows Settings.
I did this in four steps (represented roughly by the individual commits):
1. Switch from cpprest's http_client/json to Windows.Web.Http and
Windows.Data.Json
2. Switch from websocketpp to winhttp.dll's WebSocket implementation¹
3. Remove all remaining utility classes
4. Purge all dependencies from all projects and scripts on cpprest.
I also took this opportunity to add a feature flag that allows Dev
builds to run AzureConnection in-process.
¹ Windows.Networking.Sockets' API is so unergonomic that it was simply
infeasible (and also _horrible_) to use it.
## Validation Steps
I've run the Azure Connection quite a bit inproc.
Closes#4575.
Might be related to #5977, #11714, and with the user agent thing maybe #14403.
Up until now, we have been relying on the catalog signature produced for our MSIX package.
There are some things (Packaged COM, Process Explorer as of 2022) that cannot handle catalog-signed
files. It's easier and safer for us to simply sign all the executables we produce before packaging them.
Unfortunately, we can't do it before we package them. We have to unpack and re-pack our package.
In the future, this will allow us to provide a codesigned distribution that is not in an MSIX package.
TEST=Ran a build and checked out the contents of the package. They were all signed!
Closes#13294Closes#12695Closes#9670
Our Windows branch name changed, and I took this opportunity to resolve
an issue where vpack builds would occasionally fail due to GitHub rate
limiting the Azure DevOps IP addresses.
This pull request introduces a packaging phase that emits
Microsoft.Windows.Console.ConPTY, a nuget package that contains the
pseudoconsole API as well as the requisite copies of conhost.
* winconpty learned to load a version of OpenConsole.exe specific to the
processor architecture on its hosting machine
* the package, as well as its contents, is signed properly and is nearly
ready for distribution via nuget.org
* the API in conpty-static.h has been adjusted to expose
CreatePseudoConsoleAsUser and stamp out the correct DLL import/export
annotations.
* getting .NET to play right was somewhat challenging, but I tested this
against .NET 6.0 and it seemed to work properly; it shipped conpty.dll
in the right places, and it shipped OpenConsole.exe next to the
published application.
In the future, we could provide an interop assembly for C# consumers;
that is, unfortunately, out of scope today.
Closes#3577Closes#3568
Obsoletes #1130
We're now building a fully provenance, compliance, and security
validated package (vpack) through our Release pipeline. This attaches
the last phase which automates the submission into the Windows product.
It will also automatically trace back the source, commit SHA, and build
to the submission here from the Windows side.
* [x] Automates a manual activity I performed a few times recently
* [x] I work here
* [x] Ran a test of it against `release-1.12` and it worked
Enables a series of tasks run against our release pipeline that validate the security and compliance status of our code in an automated fashion. These checks include:
- Component Governance - (we had this one, it was moved to here) - Inventories open-source components used in our build
- PREfast - C/C++ static analysis for common code errors and exploits
- Policheck - Searches source code, comments, and text for words that could be sensitive legally, culturally, or geopolitically
- Credscan - Looks for credentials left behind in the code/documents and build output files
- BinSkim - Searches for common vulnerabilities in binaries
- CheckCFlags - Validates that compile/link flags match the policies recommended by Windows engineering for inclusion into the OS product image
- CFGCheck/XFGCheck - Validates that the CFG and/or XFG settings were enabled at compile and link time to guard against control flow attacks.
We're also required to run the SBOM one, but that was done in a separate PR and we're still pending the detectors being updated.
## References
- #11948 - Move from CFG to XFG once XFG task folks get back to me on it
- #11949 - Enable bug filing for SecComp tasks
- #11950 - Bulk process bugs filed by SecComp tasks
- #11947 - Validate SBOM when checkers come online
## Checklist
- [x] - Fixes#10735
- [x] - Fixes#908
- [x] - I work here
- [x] - If it fits, it sits.
This allows me to make the build pipeline, instead of the release
engineer, put the version number in the package name.
It also lets us sign multiple packages (if we ever produce more than
one.)
This pull request includes a localization config file that identifies
the modules we need to localize. It also moves us back to the
`Resources\LANGUAGE\Resources.resw` resource layout, but using wildcards
so that the build system can pick up any number of languages.
Dustin L. Howett
Michael Niksa