- Removes usages of the `linux-x64` package
- Some changes in the deployment directory structure
- Also fixes a problem where we crash if the graph construction tool fails instead of being graceful
Related work items: #2000411
The step to publish the external npm package has been removed already. This PR removes the external package from our sources (the publish step was pointing to it).
Related work items: #2168692
Revert "Merged PR 778558: Run internal unit tests with internal flag"
Reverted PR !778558
Reverted commit c22df9a9
Npm package for some reason started missing files under /tools/ after `--internal` flag was added.
Artifacts' infra only allows two states for drops - finalized and non-finalized. To avoid having too many unfinished drops, we make sure that finalize command is run for all drops made by BuildXL. So, even if a build failed and some content was missing from the drop, we would still finalize it. This means that a consumer cannot tell if a drop is complete or not by just looking at the drop content and/or its metadata. Some builds (like Cosine) depend on drops made by other builds, i.e., they need to know if a specific drop is good to use. Before this PR, users solved this problem by forcing build manifest creation and checking its status (the manifest is only created for complete drops). Manifest creation is somewhat flaky (because of signing service being flaky), so this workaround resulted in them having fewer successful builds.
This PR makes DropD add a special marker file to incomplete drops that consumers can check. If such a file cannot be added, as a backup signal, DropD will leave the drop in a non-finalized state.
Related work items: #2167925
Update the DumpPip Analyzer to have the `/includeStaticMembers` option to enumerate statically known sealed directory members. This is usefully when trying to figure out what files a process PIP depends on from other PIPs that produce those files. In addition the kind for each directory dependency is now included in the output.
Extracting the packages to a subfolderof the parent directory of release/public for Binskim. This is done to avoid the publishing of these extracted packages.
Without this, all messages are reported as "happening before initialization". Luckily we are not ignoring those reports so the behavior should still be "correct", but it should be an issue when matching the reports, if we ever do it
Related work items: #2167608
This PR makes BuildXL retry process pips that exit with 0xDEAD exit code when running in ADO. This can make our PR/rolling pipelines reliable.
Related work items: #2167066
Adding this script in the external stage of the pipeline to extarct the .nupkg files from the Bin/Release folder. Need to do this since BInSkim only supports .dll or .exe formats and fails to detect nupkg files .
Tested this in this pipeline.
Look for buildxl-external.pkgs.0.1.0-20240403.1
The .nupkg packages are converted to .zip format and then extracted to extractedPackages folder within the same directory.
Doing this avoids BinSkim from scanning the whole of the Out folder for binaries(.dll and .exe).
https://dev.azure.com/mseng/Domino/_build/results?buildId=27610984&view=artifacts&pathAsName=false&type=publishedArtifacts
Related work items: #2166618
Some pip validity check that requires graph happens in process builder. The problem is process builder is not required to have a graph. So, for such a check, the check should happen in pip graph builder instead; and this PR is about that.
And now, "Trusting statically declared accesses is not compatible with declaring opaque or source sealed directories" will have a proper error message.
Related work items: #2166157
The variables persist beyond the scope of the single task. This is useful to run RunBxlWithPAT in a pre-build task and then have the build task be exposed to these variables.
This change looks like a net negative on the surface (why replace a single script with two separate pieces?) but we need it to migrate the pipelines to 1ESPT using the BuildXL workflow, which will call BuildXL outside of any wrapping script
When setting up a StreamWriter for stdin in detoured process, the writer is set to auto flush. StreamWriter immediately does a write operation to the pipe (although the data is empty). However, when the pipe is closed (or is being closed), the write operation would fail and throw an exception. This can happen if the detoured process does not need any standard input, and the process terminates quickly before this pipe setup is completed.
It was not an issue before .NET 8 because when writing to a FileStream that represented a closed or disconnected pipe, the underlying operating system
error was ignored and the write was reported as successful. However, nothing was written to the pipe. Starting in .NET 8, when writing to a FileStream whose
underlying pipe is closed or disconnected, the write fails and an IOException is thrown.
See breaking changes in .NET 8: https://learn.microsoft.com/en-us/dotnet/core/compatibility/core-libraries/8.0/filestream-disposed-pipe
Related work items: #2166469
When ptrace is started there is a small overlap with the still existing interpose sandbox that makes ptrace observe some of what the interpose sandbox is doing. This is usually harmless, but this could involve interacting with the FIFO path when sending reports. This results in the FIFO path potentially becoming part of the pip fingerprint, and when bxl tries to hash it, we crash (since the FIFO file is not a real file).
Make sure we ignore these reports. The race is hard to avoid, and turning off interposing is not easily achievable either. Turning off reporting also has some undesirable side effects, like disabling debug logging on that part of the codebase (when this happens, is hard to understand why).
This PR ignores this report on managed side, so we don't slow down the process doing path comparisons. We also move some other path comparison logic (previously done on native side), to managed side as well.
Related work items: #2165745
Iman Narasamdya
Marcelo Lynch 🧉
Qi Wang
Sahiti Chandramouli
Serge Mera
Michael Pysson
TSEBuild
Pasindu Gunasekara 🍣
Oleksii Kononenko
Ben Witman
Pratik Lade
Semih Okur
