Add an 'orchestrator' and 'worker' mode to the BuildXLNugetInstaller so that the version is decided only on a single agent in a distributed build: this avoids race conditions between the agents when resolving a version.
In the future this strategy might be adopted to the configuration download itself, but for that a prerequisite is that the configuration is distributed in a versioned manner (like Nuget), so for now it is good enough to keep this logic as specific to the BuildXL installer.
This PR also adds an integration pipeline to test the installer before releasing. This pipeline exercises a straightforward flow and also the worker-orchestrator consistency when using the distributed mode.
Initial implementation of a build tools installer to be run in pipeline templates to acquire BuildXL and provide versioning through some global configuration.
This is meant to be distributed as a standalone package. This PR builds an MVP that we can roll out to some onboarding customers to try out (in experimental scenarios so far).
The RunCheckinTests part of rolling build validation simulates some worker disconnect scenarions which generate warnings and an error. This change removed those expected errors from showing up and cluttering the Azure DevOps console.
- Only build the engine that will be under test once, and then consume from the different validations via pipeline artifacts, instead of doing it once per validation
- Modularize the YAMLs by moving shared tasks to a common file
- Add some dependencies between the stages to short-circuit failures: namely, run the PTrace validation only after the public and
- The internal validation is now single machine, in favor of validating distribution with a small two-pip build at the end
- Add a validation that runs a small distributed build consisting of two pips, one of which is forced to run on a remote worker. This runs at the end of the validation
These tests were added years ago to ensure there were no bad interactions between gvfs and BuildXL. They have not been run in quite a while. Removing them to cleanup code and reduce pipeline execution time
- Uses Microsoft.Artifacts.Authentication for MSAL authentication in VssCredentialsFactory
- Does not use WAM broker because we can't easily get a parent window handle when running under buildxl
- Skips device code authentication to avoid having to display message on the console.
Related work items: #2063548
- This fixes the directory structure for the nuget package allowing us to declare it as a dependency of BuildXL.Utilities rather than manually deploying it.
- Add scripts for producing a public package for the native runtime.
- Add scripts to run unit tests under macOS.
- Created a new package here instead of re-using the existing one to avoid removing the old kext code in case we want to use it in the future. We'll have code that references the kext in dscript once I finish making the changes here but they won't be used for anything.
Related work items: #2082091
Revert "Merged PR 719250: Use Microsoft.Artifacts.Authentication for cache/drop authentication"
This reverts commit dc8fb8ff65.
Reverting change that's breaking dev cache authentication for users without VPN.
- Microsoft.Artifacts.Authentication provides a wrapper layer for MSAL authentication. This will allow us to continue to keep BXL up to date with the latest MSAL changes without having to change too much on our end other than bumping up package versions.
Related work items: #1969711, #2063548
Folding all of Collections into BXL.Utilities.Core. The recommended way to create Nuget packages is to do 1 dll per package unless the dlls are supposed to always be used together. Collections is a part of Utilities and it shares a large subset of dependents with Utilities.Core, so merging them together to reduce the amount of dlls.
Refactoring Instrumentation.Common to only contain Aria specific files, and moving all the non-Aria code into Utilities.Core instead. The motivation behind this is so that BXL released packages won't contain Aria packages which is an external pacakge. We want to have as little external packages as possible because it might cause diamond dependencies. (eg. BXL.Utilities depends on Aria1.0 and ExPkg depends on Aria1.1 and the customer pulls both in, there will be a version conflict)
PerformanceCollector has a dependency on Win32, and SandboxedProcess.cs has a dependency on PerformanceCollector. However, the only thing that's needed is the Aggregation Class which I've broken out of PerformanceCollector and into BXL.Utilities.Core.
Breaking Instrumentation.Common Dependency from BXL..Utilities.Core as it pulls in aria package. And the goal of BXL.Utilities.Core is to depend on no external packages
As a part of refactoring Detours dependencies. We will first need to break BXL.Utilities into BXL.Utilities.Core which only depends on BXL code and nothing else.
This is Phase I hence only a warning is logged when a credential is detected in an env var. Depending on the results obtained from the logging information the implementation is modified accordingly.
Added the CredentialScanner class to handle the functionality related to credscan
Created a unit test to test the functionality of the scanner with various test cases
Modified the SetEnvironmentVariables method to call the credscan method from CredentialScanner class.
Added allowList mechanism and a unit test to test that
Related work items: #1975564
Add nuget resolver signing
Add extra dummy sign sdk in FrontEnd/UnitTest to pass the tests
Downloaded and Extracted files need to be treated differently. Extract pip generate an opaques directory. I need to use a tool to enumerate extracted files. I'd like to split into another pr.
Related work items: #1958784
Add roslynanalyzers call in BuildXLSdk. Csc.exe will use roslynanalyzers and produces analyze results when enableRoslynanalyzers set to true. In Compliance Build, Gurdian will run roslynanalyzers with copyLogsOnly to copy the analyze results, which will then be used for processing and break
Related work items: #1941023
Use 'Microsoft.Cloud.InstrumentationFramework' package instead of 'Microsoft.Cloud.InstrumentationFramework.NetStd' that was deprecated.
Related work items: #1948491
```
SetSite failed for package [BuildXLVsPackage]Source: 'BuildXLVsPackageDev17' Description: Unable to cast object of type 'Microsoft.VisualStudio.CommonIDE.Solutions.CSlnUpdate' to type 'Microsoft.Internal.VisualStudio.Shell.Interop.IVsSolutionBuildManagerPrivate'. System.InvalidCastException: Unable to cast object of type 'Microsoft.VisualStudio.CommonIDE.Solutions.CSlnUpdate' to type 'Microsoft.Internal.VisualStudio.Shell.Interop.IVsSolutionBuildManagerPrivate'. at BuildXL.VsPackage.VsProject.ProjectFlavorFactory..ctor(BuildXLVsPackage package) in \.\Public\Src\IDE\VsIntegration\BuildXLVsPackageShared\VsProject\ProjectFlavorFactory.cs:line 59 at BuildXL.VsPackage.BuildXLVsPackage.<InitializeAsync>d__15.MoveNext() in \.\Public\Src\IDE\VsIntegration\BuildXLVsPackageShared\BuildXLVsPackage.cs:line 137
```
The problem is that the extension has its own definition of Microsoft.Internal.VisualStudio.Shell.Interop.IVsSolutionBuildManagerPrivate copied from Microsoft.Internal.VisualStudio.Shell.Interop.10.0.DesignTime.dll. The solution is to use the interface defined in Microsoft.Internal.VisualStudio.Interop.dll.
Changes:
- add a new NuGet dependency on `Microsoft.Applications.Events.Server` (which is the cross-plat Aria SDK)
- use that on Linux and Mac
- delete the native SDK previously used on Mac
Changes:
- fix various file name capitalization errors
- fix various nuget package name capitalization errors
- create and publish `Bond.CSharp.linux-x64` nuget package
- add `/etc` to default untracked scopes
- untrack `$HOME/.dotnet` when running the `Downloader` tool
- consistently spell `App.config`
With these changes, all of the following succeed for me in Ubuntu 20.04 WSL running in Windows 11:
- `./bxl.sh --minimal`
- `./bxl.sh --minimal --internal`
- `./bxl.sh --minimal --internal --shared-comp`
Once the changes make it to LKG, the next step will be to set up a Linux pipeline to build minimal selfhost. Later, that pipeline can be expanded to run unit tests etc.
Integrate BuildXL with AnyBuild SDK.
The design is to isolate AnyBuild use so that the `#if FEATURE_ANYBUILD_PROCESS_REMOTING` doesn't need to appear everywhere.
This PR depends on AnyBuild.SDK package that should be published after this PR !644510 is pushed.
Related work items: #1907310
BuildXL Release pipeline currently contains two CloudTest tasks.
Task Cf runs on CloudTest Int, while task Cg runs on CloudTest Prod.
Due to limitations in CloudTest UI, this means the results of the Cg task are not populated in the `1ES Test` tab of Azure DevOps.
To fix this, there were two approaches: either move Cf to Prod or move Cg to Int.
Moving Cf to Prod is more complicated, since there are build pipeline certificates installed and used specifically by CloudTest Int environment, which are not available on Prod.
We are taking the approach of moving Cg task to CloudTest Int. To do this, the SKU and Image used to run the Cg task needed to change, as the currently configured ones are not available on Int environment.
Here's a successful run that uses this configuration on CloudTest Int environment: https://cloudtest.visualstudio.com/CloudTest/_build/results?buildId=57793&view=results
This is part 1 of two changes required for this to work. After this PR completes, the release pipeline task and the CloudTest Validation(Gvfs) pipelines need to be edited to use the correct CloudTest configuration, which involves the following changes.
- Use Int environment instead of Prod
- Use wus2-default stamp
- Use cloudtest tenant instead of BuildXL
- Remove ComponentDetectionConverter in favor of the Microsoft.SBOM.Adapters library
- Update SBOM packages to version 2.0.99
Related work items: #1902188
Revert "Merged PR 636560: Add a helper library to extract SBOM Metadata from BuildSessionInfo to BuildX...
Add a helper library to extract SBOM Metadata from BuildSessionInfo to BuildXL.Utilities
Related work items: #1882251"
Reverted commit `c7a683ce`.
Related work items: #1882251
Marcelo Lynch
Marcelo Lynch 🧉
Michael Pysson
Pasindu Gunasekara 🍣
Will Li
Sahiti Chandramouli
David Federman
Julian Bayardo
Qi Wang
Aleksandar Milicevic
Semih Okur
Sergey Tepliakov
Chad Su
Iman Narasamdya
Pratik Lade
Oleksii Kononenko