This is the second PR that updates the dependencies in order to gert rid the dependencies to the stale keyvault nuget packages.
Related work items: #2011942
This is the first step towards fixing the Component Governance issue automatically created because of our dependency to a deprecated Azure.KeyVault package.
Related work items: #2011942
Migrate to the latest C# compiler that supports C# 11
The PR adds 'RequiredMemberAttribute' to support 'required' members from C# 11.
The latest compiler changed the behavior for '<<' operator making it checked. This PR addresses this by using 'unchecked' in the cases where the overflow is possible.
Related work items: #2017950
This PR adds .NET 7 RC1 support to BuildXL and Cache Service.
Please note, that the PR will be failing until the new LKG with net7 nuget support will be deployed.
Related work items: #1990373
This PR updates RocksDb packages to 20221219.1. The new version uses different logic for tracking the lifetime of logging adapter that prevents crashes due to inconsistent lifetime.
The original allowed having different lifetime for `RocksDbStore` (the actual wrapper around native rocksdb store) and `RocksDbLoggingAdapter` that was reponsible for marshaling lagging invocations between the native and the managed worlds.
Due to a race condition (that will be fixed separately) it was possible to leave `RocksDbStore` instance undisposed. That was cuasing runtime crashes because an undisposed rocksdb wrapper was trying to use finalized `RocksDbLoggingAdapter`.
In the design, the store tracks the lifetime of the logging adpater and the adapter itself is now hidden from the public API making the lifetime issues impossible.
This tool has been frequently failing during PR builds with:
```
##[error]DX0064 [Pip5DE6E748056C4CEF, BuildXL.Deployment - NugetPackages.winX64 - NuGet.exe [{configuration:"debug"}], d:\dbs\el\bxlint\Public\Src\Deployment\NugetPackages.dsc] - failed with exit code 57005,
##[error]Unhandled Exception: System.Threading.AbandonedMutexException: The wait completed due to an abandoned mutex.
##[error] at System.Threading.WaitHandle.ThrowAbandonedMutexException()
##[error] at System.Threading.WaitHandle.InternalWaitOne(SafeHandle waitableSafeHandle, Int64 millisecondsTimeout, Boolean hasThreadAffinity, Boolean exitContext)
##[error] at System.Threading.WaitHandle.WaitOne(TimeSpan timeout, Boolean exitContext)
##[error] at NuGet.Common.Migrations.MigrationRunner.Run() in D:\a\_work\1\s\src\NuGet.Core\NuGet.Common\Migrations\MigrationRunner.cs:line 33
##[error] at NuGet.CommandLine.Program.Main(String[] args) in D:\a\_work\1\s\src\NuGet.Clients\NuGet.CommandLine\Program.cs:line 65
```
This is an attempt to fix it.
This is Phase I hence only a warning is logged when a credential is detected in an env var. Depending on the results obtained from the logging information the implementation is modified accordingly.
Added the CredentialScanner class to handle the functionality related to credscan
Created a unit test to test the functionality of the scanner with various test cases
Modified the SetEnvironmentVariables method to call the credscan method from CredentialScanner class.
Added allowList mechanism and a unit test to test that
Related work items: #1975564
Revert "Merged PR 685164: Forward rocksdb logs to Kusto
Forward rocksdb logs to Kusto
Related work items: #1996361"
Reverted commit `60bffd1f`.
Related work items: #1996361
On newer kernel, copyfilerange no longer works cross-device. This is a known regression in the kernel.
The current BuildXL package has the Linux sandbox that doesn't contain this copyfilerange patch. AnyBuild requires this patch as it now has an updated Linux image for its agents. As a workaround, AnyBuild currently relies on two packages from BuildXL, i.e., buildxl package itself and runtime.ubuntu-linux-x64.buildxl. The latter is produced in an adhoc way.
With this PR, we can now rely solely on buildxl package.
This PR updates RuntimeContracts to 0.4.0 with the following important changes:
* All the assertions propagate caller expressions as string to make assertions more descriptive. For instance, `Contract.Requires(x != null)` will automatically embed `x != null` into a message.
* All the assertions support new interpolated string from C# 10 and allow lazy message construction. It means that now we don't need to use 'FluentAPI' and just create custom message with interpolated string syntax.
The following code is very efficient: `Contract.Assert(x > 0, $"x = {x}")`. The efficiency comes from the fact that the message is constructed only if the assertion is false and the exception will be thrown.
This PR also fixes cases when the messages were created eagerly causing excessive memory allocations that are now gone.
Related work items: #2007282
The custom loader is brittle that it hardcodes the assemblies to load. Moreover, the existing approach does not work with dotnetcore runtime due to some complicated way of MsBuild SDK resolution etc.
MsBuild team recommends that we should use Microsoft.Build.Locator for this purpose.
Because the locator can pin the MsBuild location if explicitly specified, all dotnetcore/full-framework unit tests work without any modification..
Related work items: #2006579
This change makes bxl consume an Ubuntu-built sandbox as a stop gap for the timeouts we are experimenting. The sandbox is the result of building the current prod bits (0.1.0-20221020.0.2) but changing the pool we use to an Ubuntu one.
Due to (possibly) kernel bug, copy_file_range no longer works when the file descriptors are not mounted on the same filesystems, despite what is said in the manual https://man7.org/linux/man-pages/man2/copy_file_range.2.html.
This bug breaks AnyBuild virtual filesystem (VFS) because the source file will be in the read-only (lower) layer of overlayfs, and this layer is mounted on AnyBuild FUSE, and the target file will be in the writable (upper) layer of overlayfs.
Without this PR AnyBuild cannot update its Linux image.
Related work items: #2000792, #2000797
Implement `IsCopyOnWriteSupportedByEnlistmentVolume` for Windows ReFS drives
Currently this will be gated under environment variable `EnableCopyOnWriteWin`. We intend to enable by COW for ReFS by default once we have more real-world data from QuickBuild dogfooding this feature.
Add nuget resolver signing
Add extra dummy sign sdk in FrontEnd/UnitTest to pass the tests
Downloaded and Extracted files need to be treated differently. Extract pip generate an opaques directory. I need to use a tool to enumerate extracted files. I'd like to split into another pr.
Related work items: #1958784
Renames the "Orchestrator" project to "AdoBuildRunner" to avoid confusion with the other uses of "Orchestrator" for distributed builds, and includes the tool in the BuildXL deployment. Also some minor tweaks to the tool + a test mode for connectivity tests between agents
Related work items: #1977690
This PR adds support for using the Blob Storage SDK V12 and adds support for Blob Batch operations. Then uses those to implement bulk pinning in the blob L3.
Make sure we produce the same special sandbox event on both Windows and Linux when a source rewrite is detected. Enable related unit tests for Linux as well.
Add roslynanalyzers call in BuildXLSdk. Csc.exe will use roslynanalyzers and produces analyze results when enableRoslynanalyzers set to true. In Compliance Build, Gurdian will run roslynanalyzers with copyLogsOnly to copy the analyze results, which will then be used for processing and break
Related work items: #1941023
Reverting the revert, plus handling interactive auth for local builds:
* build.cmd 'primes' the auth token cache by launching the specified cred provider (for internal+interactive builds)
* The cred provider used by bxl is allowed to read/write to the auth token cache. This implies dealing with the IsRetry pattern.
I'll have somebody (besides me) check that the local case works before merging.
Same commit as before plus:
* Force check the nuget package fp on disk (apparently this was off by default because dotnet core didn't support nuget at that time) (that should fix the DFA we saw, which is a layout difference between a public vs internal package when an agent gets reused)
* Capture stderr when executing the cred provider, for better error diagnostics.
From the issues we saw, there is still one that I was not able to repro (https://dev.azure.com/mseng/Domino/_build/results?buildId=17296398&view=logs&j=cfa20e98-6997-523c-4233-f0a7302c929f&t=144c797d-20c9-5763-9672-cca1a82cd366). I'll keep trying, but the stderr should be able to tell us more if that happens again. Hopefully we can spot this before I merge this.
Revert "Merged PR 661589: Reapply: nuget resolver schedules real pips"
This reverts commit 53d8270164.
This change is still causing some breaks in downstream validation
Reverting the revert to merge the original PR: eebb9c136a?refName=refs%2Fheads%2Fdev%2Fsmera%2Frelease509.1.1
Plus, a couple minor tweaks to make this work for linux and mac. Those changes are grouped in a separate commits so they are easier to spot.
In parallel I'm working on the rolling build cred provider configuration changes so this PR can pass.
Revert "Merged PR 659816: Nuget resolver schedules real pips
Purpose: to unblock rolling builds.
This PR changes the NuGet resolver logic so instead of downloading packages at evaluation time, download pips are scheduled so the package download and extraction happens as part of the build. The high-level flow is the following:
- The nuget resolver downloads enough of each NuGet package just to understand its layout (usually <5k) + the nuspec file. This is taking ~10s on my machine. The inspection process is still using the local + bxl cache caching layer, in the same way the fully downloaded packages were using before. So hits are also possible.
- The spec generation process now generates calls into the (internal) nuget downloader SDK. The specs are still generated on disk. We can consider doing this in memory (as other resolvers do), but that can be left for a second iteration.
- A new nuget downloader tool is now deployed as part of bxl binaries, with a small DScript SDK that schedules it. The generated specs use it. Nuget.exe is not used anymore, the nuget api is used instead. This makes the whole process less prone to machine-depending state that nuget.exe reads by default and doesn't allow us to block.
Related work items: #1942615"
Reverted commit `eebb9c13`.
Related work items: #1942615
This PR changes the NuGet resolver logic so instead of downloading packages at evaluation time, download pips are scheduled so the package download and extraction happens as part of the build. The high-level flow is the following:
- The nuget resolver downloads enough of each NuGet package just to understand its layout (usually <5k) + the nuspec file. This is taking ~10s on my machine. The inspection process is still using the local + bxl cache caching layer, in the same way the fully downloaded packages were using before. So hits are also possible.
- The spec generation process now generates calls into the (internal) nuget downloader SDK. The specs are still generated on disk. We can consider doing this in memory (as other resolvers do), but that can be left for a second iteration.
- A new nuget downloader tool is now deployed as part of bxl binaries, with a small DScript SDK that schedules it. The generated specs use it. Nuget.exe is not used anymore, the nuget api is used instead. This makes the whole process less prone to machine-depending state that nuget.exe reads by default and doesn't allow us to block.
Related work items: #1942615
This only updates the Windows packages. The macOS and linux packages need to be created by building from source. We do not perform distributed builds on those platforms so I don't expect a compatibility issue with this change. We are planning to upgrade everything to protobuf anyway.
Related work items: #1938788
Changes:
- fix various file name capitalization errors
- fix various nuget package name capitalization errors
- create and publish `Bond.CSharp.linux-x64` nuget package
- add `/etc` to default untracked scopes
- untrack `$HOME/.dotnet` when running the `Downloader` tool
- consistently spell `App.config`
With these changes, all of the following succeed for me in Ubuntu 20.04 WSL running in Windows 11:
- `./bxl.sh --minimal`
- `./bxl.sh --minimal --internal`
- `./bxl.sh --minimal --internal --shared-comp`
Once the changes make it to LKG, the next step will be to set up a Linux pipeline to build minimal selfhost. Later, that pipeline can be expanded to run unit tests etc.
Component governance still complains about the NPM in node v17.1.0. I upgrade the node to v17.2.0, which has just released yesterday. That one contains the npm 1.8.4.
The main work was done by @<Pasindu Gunasekara 🍣> and I was integrating the new approach into our sdk.
This PR adds `LogGenerator` project that provides C# `SourceGenerator` for creating logs.
The benefit of this approach is that we don't need another compiler invocation and separate dlls are not produced if the project has generated logs. Instead those logs are produced at one step. So it should reduce the size of the output directories and speed up the self host build.
We don't change the way we generate logs, instead the solution is based on the existing log gen logic but it is called by the comipler.
**Considerations**
To make the transition simpler we now support two ways of generating logs via `generateLogs` argument and `generateLogsInProc`. The first one is the old one and the second one is the new one.
Also there is an environment variable that allows switching the mode globally from the old one to the new one.
Revert "Merged PR 636560: Add a helper library to extract SBOM Metadata from BuildSessionInfo to BuildX...
Add a helper library to extract SBOM Metadata from BuildSessionInfo to BuildXL.Utilities
Related work items: #1882251"
Reverted commit `c7a683ce`.
Related work items: #1882251
This should eliminate this warning in our various validation builds
`The file 'd:\dbs\el\bxlint\.CloudBuild\DominoDropConfig.json' is being used as a source file, but is not under a defined mountpoint. This file is thus 'untracked', and changes to it will not impact incremental builds`
- Update MSVC package to 14.16.27034
- Update internal feed to use VisualCppTools.Internal.VS2017Layout from devdiv feed.
- Update external build instructions to get user to download visual studio build tools manually.
- Add Qspectre flag to msvc.
- Ignore some newer warnings being hit on windows sdk source files.
Related work items: #1846018
The new packages contain updated native binaries for Linux which are compatible with a broader range of Linux distros.
The RocksDB package was updated in !617830
The core change is in `AzureEventHubClient.cs`, where we can now authenticate with an EventHub instance using a Managed Identity.
Related work items: #1826212, #1827238
Expose the analyzers support from the Bxl sdk.
Bxl supported Roslyn analyzers for a long time, but the list of supported analyzers was built in into Bxl Sdk.
This PR exposes `analyzers` properties and every project may add extra analyzers if needed:
```
@@public
export const dll = BuildXLSdk.library({
assemblyName: "BuildXL.Cache.ContentStore.Distributed",
sources: globR(d`.`,"*.cs"),
analyzers: [importFrom("protobuf-net.BuildTools").pkg]});
```
When moving or removing directory, Detours needs to enumerate it to report file writes on its members. During enumeration the path can go beyond MAX_PATH.
Tested on failed QuickBuild repo.
Related work items: #1836781
To get an absolute path for a file descriptor, we used to call `readlink("/proc/self/fd/{fd}")`. That's perfectly fine and still the best way to do it; the problem is that it is slow. (for example, it is not uncommon that a C compiler calls `putc` millions of times to produce a build output).
This PR implements a file descriptor table where where a mapping from a file descriptor to an absolute path is cached.
A new entry is computed and inserted into the table whenever a path for a file descriptor is needed but it hasn't already been computed.
Because file descriptors may be reused for different files over time, whenever a file descriptor is closed (i.e., the `close` syscall is intercepted) the corresponding entry in the table is invalidated.
Related work items: #1833259
Consider the following scenario:
1. Absent probe on path A/B/C/file.txt
2. Reparse point created on C in path A/B/C, where now A/B/C -> D/E/F and D/E/F/file.txt exists.
3. Present probe on path A/B/C/file.txt (by virtue of 2)
Step 1) populates the reparse point cache with A/B/C/file.txt as not needing resolution (since the path is absent). 2) should invalidate A/B/C, and now A/B/C does need resolution because C is a reparse point. But A/B/C/file.txt does not get invalidated, and therefore when 3) happens, A/B/C/file.txt is not resolved. This manifests in a cache miss when file.txt is a produced output, since we don't compensate for the absence of the file on cache lookup when the unresolved path is stored.
This PR adds:
* A conservative invalidation on reparse point creation (we don't detour the ioctl call, but we see a handle being opened for write and reparse point flags as the immediately previous operation). We invalidate the path in that case to stay on the safe side.
* A tracking mechanism for all descendants of a given path that the reparse point cache knows about. When invalidating a path, all descendants stored in the cache are also invalidated. This accounts for the above scenario, where when C is turned into a reparse point, both A/B/C and A/B/C/file.txt are invalidated
* A lightweight C++ unit test infrastructure (using boost) with unit tests for the new structure.
Related work items: #1824112
This PR updates the version of ErrorProne.NET CoreAnalyzers that now detects an unobserved calls to `CancellationToken.Register`.
It is important to observe and dispose the `CancellationTokenRegistration` instance when they obtained from non-local `CancellationToken`s because otherwise we may ended up with a memory leak when a longer-lived object will hold references to a transient closures created by the callbacks provided to `CancellationToken.Register`.
It seems that the hack I've added to avoid allocations is not needed.
One of the NLog maintainers opened a bug: https://github.com/microsoft/BuildXL/issues/1283 with the explanation of the right solution and this PR uses the proposed solution instead of relying on a custom target I've created previously.
In my little spare time, I found a chance to improve AsyncFixer not to forget my coding skills. Here are changes needed to upgrade to v1.5.1:
- Fixed false negatives, especially for 01:UnnecessaryAsync. That's why I had to remove many redundant async/await keywords. It is always faster to directly return the Task. We rarely need async/await keywords in redundant cases due to some special exception handling. However, we have suppressed "01:UnnecessaryAsync" in only one case at our codebase so far.
- Fixed false positives for other analyzers, especially for 02:BlockingCallInsideAsync. I removed many suppressions, which made me pretty happy.
I am still OOF, but I did not want to postpone these changes. I have also open-sourced AsyncFixer. Feel free to contribute: https://github.com/semihokur/asyncfixer
@<Sergey Tepliakov>, there are several cases where we use async/await keywords with the new using statement syntax. You reported that at some point as far as I remember. I improved AsyncFixer to detect cases like below so that it does not suggest removing async/await
```
async Task foo()
{
using var stream = new MemoryStream();
int streamOperation()
{
return stream.Read(null);
}
Task t = Task.Run(() => streamOperation());
await t;
}
```
@<John Erickson>, could you please point me to the artifact services projects where AsyncFixer is enabled? I can upgrade AsyncFixer in those projects as well.
This PR adds a facility to bxl sdk to use source generators and also it adds one source generator that can generate struct-based record-like behavior as well as two other generators that can:
* Generate ToString implementation similar to records ToString behavior but with the ability to fine tune the final ToString impl.
* Generate equality members.
Related work items: #1805964
Remove TransientFaultHandling completely. Polly has been tested for a while now and this should simplify our dependencies.
The only trace of it left is that ADO packages depend on it.
Related work items: #1807652
This PR adds support for .NET 5.
The only current limitation is that QTest is disabled in .NET 5 because it was failing with some "can't find the right assembly" issue that I'm going to investigate.
Also, the default qualifier stays the same and I need some help testing the changes on the mac and on Linux.
This PR also allow us using C# 9 features like records **for all target frameworks** and the following code compiles just fine:
```csharp
public record X
{
public int Y { get; init; }
}
```
Keep in mind, that you need the most recently released VS in order to see intellisense for this code.
This adds support for LZ4, Zlib and Zstd compression to our RocksDb binaries on Windows. Current compression support is:
- Linux supports lz4 and snappy
- Mac supports lz4 and snappy
- Windows supports lz4, zlib, zstd and snappy
We can build in more if we need. I only added these to the Windows binaries because it's what we'll be using in the datacenter.
Related work items: #1790176
This is the second PR that is required to make the final PR for adding .net5 support simpler.
The PR updates the compiler to a 3.7 version (this version does not support C# 9 yet), and fixes a bunch of nullability warnings that appear after the upgrade to 3.7 and some of them will be visible only when targeting .net 5.
This is another low risk PR.
There is a desire to move to Polly, since TransientFaultHandling is not compatible with regular NuGet when using netstandard.
This adds the capability to put this under a flag to start testing the transition.
Related work items: #1791278
Michael Pysson
Sergey Tepliakov
Julian Bayardo
Marcelo Lynch 🧉
Sahiti Chandramouli
Qi Wang
Pasindu Gunasekara 🍣
Semih Okur
Iman Narasamdya
Serge Mera
Christian Castaneda
Aleksandar Milicevic
Lance Collins
Oleksii Kononenko
Rijul Luman
Kevin Bazan Williams
Juan Carlos Guzman Islas
Nora Huang
