Component governance still complains about the NPM in node v17.1.0. I upgrade the node to v17.2.0, which has just released yesterday. That one contains the npm 1.8.4.
The main work was done by @<Pasindu Gunasekara 🍣> and I was integrating the new approach into our sdk.
This PR adds `LogGenerator` project that provides C# `SourceGenerator` for creating logs.
The benefit of this approach is that we don't need another compiler invocation and separate dlls are not produced if the project has generated logs. Instead those logs are produced at one step. So it should reduce the size of the output directories and speed up the self host build.
We don't change the way we generate logs, instead the solution is based on the existing log gen logic but it is called by the comipler.
**Considerations**
To make the transition simpler we now support two ways of generating logs via `generateLogs` argument and `generateLogsInProc`. The first one is the old one and the second one is the new one.
Also there is an environment variable that allows switching the mode globally from the old one to the new one.
Revert "Merged PR 636560: Add a helper library to extract SBOM Metadata from BuildSessionInfo to BuildX...
Add a helper library to extract SBOM Metadata from BuildSessionInfo to BuildXL.Utilities
Related work items: #1882251"
Reverted commit `c7a683ce`.
Related work items: #1882251
This should eliminate this warning in our various validation builds
`The file 'd:\dbs\el\bxlint\.CloudBuild\DominoDropConfig.json' is being used as a source file, but is not under a defined mountpoint. This file is thus 'untracked', and changes to it will not impact incremental builds`
- Update MSVC package to 14.16.27034
- Update internal feed to use VisualCppTools.Internal.VS2017Layout from devdiv feed.
- Update external build instructions to get user to download visual studio build tools manually.
- Add Qspectre flag to msvc.
- Ignore some newer warnings being hit on windows sdk source files.
Related work items: #1846018
The new packages contain updated native binaries for Linux which are compatible with a broader range of Linux distros.
The RocksDB package was updated in !617830
The core change is in `AzureEventHubClient.cs`, where we can now authenticate with an EventHub instance using a Managed Identity.
Related work items: #1826212, #1827238
Expose the analyzers support from the Bxl sdk.
Bxl supported Roslyn analyzers for a long time, but the list of supported analyzers was built in into Bxl Sdk.
This PR exposes `analyzers` properties and every project may add extra analyzers if needed:
```
@@public
export const dll = BuildXLSdk.library({
assemblyName: "BuildXL.Cache.ContentStore.Distributed",
sources: globR(d`.`,"*.cs"),
analyzers: [importFrom("protobuf-net.BuildTools").pkg]});
```
When moving or removing directory, Detours needs to enumerate it to report file writes on its members. During enumeration the path can go beyond MAX_PATH.
Tested on failed QuickBuild repo.
Related work items: #1836781
To get an absolute path for a file descriptor, we used to call `readlink("/proc/self/fd/{fd}")`. That's perfectly fine and still the best way to do it; the problem is that it is slow. (for example, it is not uncommon that a C compiler calls `putc` millions of times to produce a build output).
This PR implements a file descriptor table where where a mapping from a file descriptor to an absolute path is cached.
A new entry is computed and inserted into the table whenever a path for a file descriptor is needed but it hasn't already been computed.
Because file descriptors may be reused for different files over time, whenever a file descriptor is closed (i.e., the `close` syscall is intercepted) the corresponding entry in the table is invalidated.
Related work items: #1833259
Consider the following scenario:
1. Absent probe on path A/B/C/file.txt
2. Reparse point created on C in path A/B/C, where now A/B/C -> D/E/F and D/E/F/file.txt exists.
3. Present probe on path A/B/C/file.txt (by virtue of 2)
Step 1) populates the reparse point cache with A/B/C/file.txt as not needing resolution (since the path is absent). 2) should invalidate A/B/C, and now A/B/C does need resolution because C is a reparse point. But A/B/C/file.txt does not get invalidated, and therefore when 3) happens, A/B/C/file.txt is not resolved. This manifests in a cache miss when file.txt is a produced output, since we don't compensate for the absence of the file on cache lookup when the unresolved path is stored.
This PR adds:
* A conservative invalidation on reparse point creation (we don't detour the ioctl call, but we see a handle being opened for write and reparse point flags as the immediately previous operation). We invalidate the path in that case to stay on the safe side.
* A tracking mechanism for all descendants of a given path that the reparse point cache knows about. When invalidating a path, all descendants stored in the cache are also invalidated. This accounts for the above scenario, where when C is turned into a reparse point, both A/B/C and A/B/C/file.txt are invalidated
* A lightweight C++ unit test infrastructure (using boost) with unit tests for the new structure.
Related work items: #1824112
This PR updates the version of ErrorProne.NET CoreAnalyzers that now detects an unobserved calls to `CancellationToken.Register`.
It is important to observe and dispose the `CancellationTokenRegistration` instance when they obtained from non-local `CancellationToken`s because otherwise we may ended up with a memory leak when a longer-lived object will hold references to a transient closures created by the callbacks provided to `CancellationToken.Register`.
It seems that the hack I've added to avoid allocations is not needed.
One of the NLog maintainers opened a bug: https://github.com/microsoft/BuildXL/issues/1283 with the explanation of the right solution and this PR uses the proposed solution instead of relying on a custom target I've created previously.
In my little spare time, I found a chance to improve AsyncFixer not to forget my coding skills. Here are changes needed to upgrade to v1.5.1:
- Fixed false negatives, especially for 01:UnnecessaryAsync. That's why I had to remove many redundant async/await keywords. It is always faster to directly return the Task. We rarely need async/await keywords in redundant cases due to some special exception handling. However, we have suppressed "01:UnnecessaryAsync" in only one case at our codebase so far.
- Fixed false positives for other analyzers, especially for 02:BlockingCallInsideAsync. I removed many suppressions, which made me pretty happy.
I am still OOF, but I did not want to postpone these changes. I have also open-sourced AsyncFixer. Feel free to contribute: https://github.com/semihokur/asyncfixer
@<Sergey Tepliakov>, there are several cases where we use async/await keywords with the new using statement syntax. You reported that at some point as far as I remember. I improved AsyncFixer to detect cases like below so that it does not suggest removing async/await
```
async Task foo()
{
using var stream = new MemoryStream();
int streamOperation()
{
return stream.Read(null);
}
Task t = Task.Run(() => streamOperation());
await t;
}
```
@<John Erickson>, could you please point me to the artifact services projects where AsyncFixer is enabled? I can upgrade AsyncFixer in those projects as well.
This PR adds a facility to bxl sdk to use source generators and also it adds one source generator that can generate struct-based record-like behavior as well as two other generators that can:
* Generate ToString implementation similar to records ToString behavior but with the ability to fine tune the final ToString impl.
* Generate equality members.
Related work items: #1805964
Sergey Tepliakov
Julian Bayardo
Semih Okur
Pasindu Gunasekara 🍣
Marcelo Lynch 🧉
Oleksii Kononenko
Serge Mera
Rijul Luman
Michael Pysson 🍕
Aleksandar Milicevic
Kevin Bazan Williams
Iman Narasamdya
Juan Carlos Guzman Islas