Currently we only catch / log only InvalidArgumentException that we throw when user provided incorrect arguments, and we rely on individual analyzers to catch any other exceptions that occur during their invocation. We've observed cases where the unobserved exceptions were not properly handled by the calller of an analyzer. This PR expands the outer try/catch, so we always process / log the exception (mainly to facilitate future debugging).
Related work items: #2116741
When opening a file by file id, Detours needs to know the path corresponding to the file id. Previously, we open a handle with `GENERIC_READ`. However, with that desired access, opening a remote file will fetch the file from remote unnecessarily.
This PR simply replaces `GENERIC_READ` with `FILE_READ_ATTRIBUTES`, which is sufficient for getting the path.
Additional changes:
- Instead of hardcoding the file attributes, we use the user specified file attributes when opening a handle for `GetFinalPathByHandle`.
- Refactor network UT and add a disclaimer that the UT can time out.
Related work items: #2125608
- Make the public validation public end-to-end, by building a public engine and then using that to run tests (also without passing `--internal`)
- Use the 1ESPT build job for the distributed build test. We're still keeping the old YML for backup, but it becomes unused.
- Note that using 1ESPT for our own selfhost is not possible due to how we invoke `bxl.sh`, that's why that one still adds the worker stage manually.
Addresses some PR feedback from !742780. Points were mostly around documentation and DScript tweaks. I didn't want to rev on that PR since it was already a challenge to get merged with so many approvals.
This pull request includes baselines **with an expiration date of 180 days from now** automatically generated for your 1ES PT-based pipelines. Complete this pull request as soon as possible to make sure that your pipeline becomes compliant. Longer delays in completing this PR can trigger additional emails or S360 alerts in the future.
1ES PT Auto-baselining feature helps capture existing violations in your repo and ensures to break your pipeline only for newly introduced SDL violations after baselining. Running SDL tools in break mode is required for your pipeline to be compliant. Go to https://aka.ms/1espt-autobaselining for more details.
Allow to specify a list of relative paths to exclude from point nuget packages. This largely simplifies deployment problems when multiple nuget packages include the same dll (many times with different versions).
Added the pipSpecificProperty - pipFingerprintingSalt to pass the pipSemistableHash value and the salt value to salt a pip.
Refactored some code related to handling of pipSpecificProperties;
Ensure the data is shared between orchestrator and workers in a distributed build.
Added a unit test to tests end to end working of this feature and also this covers the IS part. Added an Engine based test to see if the graph has been recomputed or not.
Added a unit to check computation of static and content fingerprint.
This PR adds support for [PublicAPIAnalyzers](https://github.com/dotnet/roslyn-analyzers) - the analyzers developed by the Roslyn team and used by .NET Community to enforce the public surface stability of libraries.
The idea behind the analyzer is that during the build, the analyzer checks `PublicAPI.Shipped.txt` and `PublicAPI.Unshipped.txt` files for the content of the project's public surface. And if there are changes in the public surface in respect to those files, the analyzer would emit a warning.
The analyzers won't prevent you from making breaking changes, but those breaking changes would be intentional (plus some of them are not obvious at all, for instance, adding a default parameter is a runtime breaking change, even though this is not a compile time breaking change).
In order to enable the analyzer the two steps are required:
1. Add `usePublicApianalyzer: true` to a project spec
2. Add `publicApiFiles` to a project spec.
The last part is required since there are multiple strategies that can be used to force the API compatibility. For instance, the API surface might be different for different target frameworks, in this case a helper function `gbetFrameworkSpecificPublicApiFiles` can be used that will get different files depending on the target framework.
The analyzer also warn on some public API anti-patterns, like default arguments for methods with overloads or for default arguments in general.
This PR also enables such analysis for cache hashing project, since the stability of that project is very important in order to avoid runtime failures in BuilxXL, Arficats and CloudBuild.
Related work items: #2112018
Use the ado runner cache config generation for our selfhost Linux validations:
* Remove the explicit generation of a cache config for the ado build runner (distributed) case
* Use the new ado build runner functionality and let it generate one (ephemeral, datacenter wide, as we are currently dogfooding)
* Make internal build distributed (main reason is to keep dogfooding the ephemeral cache with a more real-life build). The explicit distributed clean build with a 2-pip build remains as is.
There is already a check for the HistoricMetadataCache not being initialized, but the object used to get to the HistoricMetadataCache may also be uninitialized and BuildXL is still crashing in that case.
Related work items: #2124645
Revert the LogAction changes in BuildXL.Processes.
Remove the UnexpectedCondition log function call because it's from BuildXL.Tracing.
Assert the package dependencies of BuildXL.Processes.
Related work items: #2091292
This PR adds logging of a final config object that is used by bxl during a build. Config serialized into a json and placed into a separate file in a logs directory.
Related work items: #2119573
This pr cleans the reference of LogGen. BuildXLSdk will not add BuildXL.Tracing reference for generateLog.
Add an arg to control AriaCommon reference for LogGen. LogGen don't need AriaCommon if generate local only logger.
- Only build the engine that will be under test once, and then consume from the different validations via pipeline artifacts, instead of doing it once per validation
- Modularize the YAMLs by moving shared tasks to a common file
- Add some dependencies between the stages to short-circuit failures: namely, run the PTrace validation only after the public and
- The internal validation is now single machine, in favor of validating distribution with a small two-pip build at the end
- Add a validation that runs a small distributed build consisting of two pips, one of which is forced to run on a remote worker. This runs at the end of the validation
Using the old behavior (not calling this method) will result in less fidelity logged for the binary logger during evaluation. The new behavior requires ALL loggers to opt-in.
* Fixed typo in type.
* Remove unnecessary new constructor in `SandboxedProcessInfo`.
* Remove unnecessary log action in `SandboxExec` since it doesn't listen to any source.
(Somehow, I have a feeling that the AI-generated description will provide a much better description.)
- Reruns the artifacts cred provider with the -I argument if the credential provided by the artifacts cred provider is invalid for npm.
- Not necessary for Linux builds right now because they will always return a PAT, and nuget should call the cred provider with -I when necessary.
Related work items: #1969322
[Blob L3 GC] Enable deleting old untracked containers by default. This has been running successfully for a couple of days in the cbrmtest-centralus cache, which is the main cache where we've been doing experiments. Enabling for all other caches. This keeps the option of disabling via config.
When there is a low disk space, we cancel PipQueue and running pips by triggering SchedulerCancellationToken. The pips get cancelled and they have SandboxedProcessPipExecutionStatus.Canceled. However, we do not set RetryInfo for those pips, so Canceled sandboxed result is translated to PipResultStatus.Failed due to the lack of RetryInfo. The orchestrator receives pip results with Failed with no error logs, so we log DistributionPipFailedOnWorker errors.
We should have set RetryInfo for those cancelled pips. Because we check environment.Context.CancellationToken instead of SchedulerCancellationToken, we skip setting RetryInfo. Context.CancellationToken is triggered when CTRL-C is pressed. SchedulerCancellationToken is triggered when we request termination in Scheduler.
Related work items: #2121638
Some internal errors happen when interactions with external systems are badly degraded. One example is that cache operations may be taking extremely long to complete. Since cancellation tokens do not transit through the entire cache layer, the above requested cancellation may not actually take effect for upwards of hours. In these cases, the build essentially still hangs until externally cancelled.
To avoid this, add a timeout for the amount of time the PipQueue draining thread will wait for the completion of pending tasks upon cancellation. This might leave the application in a weird state, but it gives a better shot of shutting down and flushing telemetry in a timely manner.
Related work items: #2108369
* Adds the machinery to deal with CLI args at the ado build runner level by reusing the parser the main bxl app uses and follow a similar 'configuration object' pattern. This introduces the ability to pass an end of caller arguments mark '--' (a la JS) to split caller arguments from forwarding arguments.
* Adds cache config related parameters. Populate the cache config generation configuration object, with sensible defaults. When the cache config is generated, it is inserted as the first forwarding argument.
TODOs:
* Move other relevant environment variables used today as arguments to this model.
* Tests. We need some uber-infrastructure to be able to run the runner mocking some underlying components. That will allow to add tests for this and for other functionality
Pratik Lade
Oleksii Kononenko
Julian Bayardo
CloudDev Build Account
Pasindu Gunasekara 🍣
Iman Narasamdya
Serge Mera
Andy Gerlicher
Marcelo Lynch 🧉
Michael Pysson
MerlinBot.PPE
Sahiti Chandramouli
Sergey Tepliakov
Qi Wang
Semih Okur
Dmitriy Shepelev
Juan Carlos Guzman Islas