Integrate BuildXL with AnyBuild SDK.
The design is to isolate AnyBuild use so that the `#if FEATURE_ANYBUILD_PROCESS_REMOTING` doesn't need to appear everywhere.
This PR depends on AnyBuild.SDK package that should be published after this PR !644510 is pushed.
Related work items: #1907310
BuildXL Release pipeline currently contains two CloudTest tasks.
Task Cf runs on CloudTest Int, while task Cg runs on CloudTest Prod.
Due to limitations in CloudTest UI, this means the results of the Cg task are not populated in the `1ES Test` tab of Azure DevOps.
To fix this, there were two approaches: either move Cf to Prod or move Cg to Int.
Moving Cf to Prod is more complicated, since there are build pipeline certificates installed and used specifically by CloudTest Int environment, which are not available on Prod.
We are taking the approach of moving Cg task to CloudTest Int. To do this, the SKU and Image used to run the Cg task needed to change, as the currently configured ones are not available on Int environment.
Here's a successful run that uses this configuration on CloudTest Int environment: https://cloudtest.visualstudio.com/CloudTest/_build/results?buildId=57793&view=results
This is part 1 of two changes required for this to work. After this PR completes, the release pipeline task and the CloudTest Validation(Gvfs) pipelines need to be edited to use the correct CloudTest configuration, which involves the following changes.
- Use Int environment instead of Prod
- Use wus2-default stamp
- Use cloudtest tenant instead of BuildXL
- Remove ComponentDetectionConverter in favor of the Microsoft.SBOM.Adapters library
- Update SBOM packages to version 2.0.99
Related work items: #1902188
Revert "Merged PR 636560: Add a helper library to extract SBOM Metadata from BuildSessionInfo to BuildX...
Add a helper library to extract SBOM Metadata from BuildSessionInfo to BuildXL.Utilities
Related work items: #1882251"
Reverted commit `c7a683ce`.
Related work items: #1882251
A couple of bug fixes in the SDK (the comments have more details), plus an integration test using the new API to ensure the arguments are working properly (because this gave us some trouble).
- Update MSVC package to 14.16.27034
- Update internal feed to use VisualCppTools.Internal.VS2017Layout from devdiv feed.
- Update external build instructions to get user to download visual studio build tools manually.
- Add Qspectre flag to msvc.
- Ignore some newer warnings being hit on windows sdk source files.
Related work items: #1846018
Implement a timeout for pip results to come back from workers. If after 5.25 hours we don't hear back from a pip, disconnect the worker. Also adds an integration test to exercise the behavior.
The 5.25 hour default is chosen because our longest pip timeouts are set to 5h for some queues, and this is meant to be a failover that kicks in when we lost the pip result for some reason.
Started using Span<T> where possible. For the first iteration of improvements, integrated the usage of Span<T> to the VSO0 hasher.
Unfortunately, the APIs that make this possible only exist for netcore, so an `#if` was required.
Here are the results of the improvement (meassured using BenchmarkDotNet and included other hash types for reference):
TL;DR: Memory usage for VSO0 was greately decreased and is now O(1).
Benchmark code: https://gist.github.com/JuanCarlosGI/effa599c286a1b8fcfb3f8b05e5b4cae
Before:
| Method | SizeBytes | HashType | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|------- |----------- |--------- |----------------:|--------------:|--------------:|------:|------:|------:|----------:|
| Hash | 1024 | Dedup64K | 12.82 us | 0.140 us | 0.117 us | - | - | - | 632 B |
| Hash | 1024 | SHA256 | 10.68 us | 0.204 us | 0.181 us | - | - | - | 192 B |
| Hash | 1024 | Vso0 | 15.59 us | 0.314 us | 0.385 us | - | - | - | 832 B |
| Hash | 1048576 | Dedup64K | 3,722.62 us | 73.686 us | 112.527 us | - | - | - | 5088 B |
| Hash | 1048576 | SHA256 | 4,460.21 us | 87.853 us | 86.284 us | - | - | - | 192 B |
| Hash | 1048576 | Vso0 | 4,595.16 us | 64.928 us | 57.557 us | - | - | - | 2992 B |
| Hash | 1073741824 | Dedup64K | 3,724,440.25 us | 11,676.096 us | 10,921.827 us | - | - | - | 3166336 B |
| Hash | 1073741824 | SHA256 | 4,445,747.54 us | 8,232.586 us | 7,700.766 us | - | - | - | 192 B |
| Hash | 1073741824 | Vso0 | 4,573,788.95 us | 10,057.379 us | 9,407.679 us | - | - | - | 2548032 B |
After:
| Method | SizeBytes | HashType | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|------- |----------- |--------- |----------------:|--------------:|--------------:|------:|------:|------:|----------:|
| Hash | 1024 | Dedup64K | 12.92 us | 0.255 us | 0.332 us | - | - | - | 632 B |
| Hash | 1024 | SHA256 | 10.77 us | 0.192 us | 0.228 us | - | - | - | 192 B |
| Hash | 1024 | Vso0 | 15.12 us | 0.300 us | 0.485 us | - | - | - | 560 B |
| Hash | 1048576 | Dedup64K | 3,703.92 us | 59.415 us | 49.615 us | - | - | - | 5088 B |
| Hash | 1048576 | SHA256 | 4,413.44 us | 66.234 us | 61.955 us | - | - | - | 192 B |
| Hash | 1048576 | Vso0 | 4,563.11 us | 88.906 us | 83.163 us | - | - | - | 560 B |
| Hash | 1073741824 | Dedup64K | 3,671,089.55 us | 22,499.938 us | 21,046.456 us | - | - | - | 3164944 B |
| Hash | 1073741824 | SHA256 | 4,401,292.67 us | 32,...
Add microsoft-internal Microsoft.Caching.Redis NuGet package. This is a fork of the StackExchange.Redis repo. There should be no breaking changes, other than renaming the namespace `StackExchange.Redis` to `Microsoft.Caching.Redis`
Related work items: #1822916
When testing, I had used net472 to be able to use VS to run tests manually. This time, when using BXL to test using netcoreapp3.0, I was able to debug deployment problems.
A findstr was being performed as part of PartB that was scanning the whole BuildXL root instead of the test cone. This eventually caused an OOM when the repo + outputs became large enough.
Change NodeJs version to appease Component Governance.
Internally, we rely on private repackaged NodeJS, but it has not been updated in a while. So this PR makes everyone consume the public version (from nodejs.org).
Related work items: #1766547, #1766548
Let P be a pip that has output directory D. After execution, P will have some nested directory created underneath D, say D\E.
1. Execute P, expect cache miss.
2. Modify P's input, and execute P again, expect cache miss.
3. Execute P, expect not scheduled, but it is scheduled.
The reason why P gets dirty in the 3rd build is because the deletion of D\E before P is executed in the 2nd build. D\E is a directory and is not superseded.
This change introduces a mode that can be used to supersede all subpaths, instead of file paths. To be on the safe side, the superseding of container path is only used if the change is directory deletion. If the change is rename, then superseding is not used in order to avoid orphaning tracked children.
To be effective, we try to avoid doing renaming as much as possible. This is done by forcing posix delete.
Users can go back to the legacy behavior (if problematic) using `/fileChangeTrackerSupersedeMode:FileOnly`. Or if posix delete is causing problem, then `/posixDeleteMode:RunLast`.
Related work items: #1764535
- Releasing resources before PostProcess step and just after Execute step.
- We do not choose a worker if there are available slots, but the materialize queue is full.
- PostProcess running in the CacheLookup queue instead of Materialize.
- An option to disable load-balancing among workers.
Related work items: #1707218, #1720422
Revert "Merged PR 545943: Dedup drop"
This reverts commit b1ba48cd45.
New drop is causing issues in CB. Reverting until we get a new version of drop with the fix.
- all tests in `BasicGvfsOperationTests` run against both a git and a gvfs repo
- when running against a gvfs repo, run both against an existing repo and a freshly cloned one
- when running against a gvfs repo, assert that the `.gvfs/GVFS_projection` file changed when actual source files have pending unmaterialized changes
Related work items: #1694134
Initial set of CloudTest Tests for Gvfs
To prep your machine to run locally:
1) Install git 4 windows
2) Install Gvfs
To build:
```
bxl /f:output='out\bin\cloudtest\*'
```
To run, execute the following in an **elevated** shell:
```
Out\Bin\CloudTest\debug\Gvfs\RunTests.cmd
```
The C# compiler supports a strict mode, the mode that was design to emit extra diagnostics for existing code and not break existing customers. The classical example is definitely assignment checks when all the public members are removed because the type is used via reference assemblies (we hit this case actually with `RelativePath`), or when a value type can be used for doing locks (we hit it as well).
Related work items: #1710763
Various small changes that allow building selfhost with BuildXL on Linux.
What's missing:
- downloading nugets
- remote telemetry
- bond.csharp tool (`gbc`) for Linux (needs to be built from sources and published as a new nuget)
- rush stuff
If (1) nugets are first downloaded manually, and (2) `gbc` is manually replaced with a Linux binary, building minimal selfhost on Linux works.
This PR introduces two new sandboxing modes for macOS based systems: `macOSDetours` and `macOSHybrid`. The former does process sandboxing exclusively through DYLD (dynamic library loading) injection the former uses DYLD and EndpointSecurity in combination to battle the currently lossy EndpointSecurity implementation. Having a detours like DYLD sandbox helps when testing and iterating new approaches quickly. The sandboxing layer has been reworked to support generic sandboxing events (read non kext-based sandbox reports) and hand them over to BuildXL.
Related work items: #1532977, #1669177, #1679422
This PR will finish the instrastruture needed to run tests in cloudtest from our repo.
* There is an optional CloudTest validation step in the PR. Currently called `CloudTest validation (Gvfs)`.
- If we do anything to the change detection logic we should manually run this.
* If you change anything under the CloudTest folder a sibbling validation will be automatically kicked off and is marked required called `CloudTest validation (Gvfs) - Required`
* There is a validation step in our CI between the Dogfood and Canary stages called `Cg: CloudTest validation (gvfs)`
- This one is currently marked as optional until we have enough confidence in the signal and can make it required.
The infrastructure has been set up in that it is not just for Gvfs validation. Future CloudTest tests can simply be added as a sibbling under the `private\CloudTest` folder following the gvfs tests pattern and following the user guide on `http://aka.ms/CloudTest` on how to edit the xml files.
If we add more I would recommend naming the queue and the jobs in the pr validation and CI to reflect this.
Iman Narasamdya
Pratik Lade
Marcelo Lynch 🧉
Michael Pysson
Pasindu Gunasekara 🍣
Oleksii Kononenko
Sergey Tepliakov
Juan Carlos Guzman Islas
Aleksandar Milicevic
Serge Mera
David Wright
Kristijan Simic
Semih Okur
Rijul Luman
Danny van Velzen
