Something in Office's dev build process tree may be intercepting and rendering the escape characters used by terminal hyperlink support. Reverting this until it can be debugged further to avoid ugly characters from showing up on the console
Related work items: #2211824
The remote cache cutoff feature stops querying the L3 after n number of consecutive misses (default today is n = 2).
If a pip is set to disabled cache, don't increment the UpstreamCacheMissLogestChain for its dependent pips to avoid result in a miss for the whole chain
Related work items: #2038931
Update engines.vscode to match @types/vscode. My suspicion is that newer versions of node are actually enforcing this (the error started to manifest on the new YAML release pipeline).
Original PR: !802728
After adding a number of logging statements to narrow down the issue, it had been determined that this could be a race condition with the garbageCollector in the unit test.
In the garbageCollector the concurrent map called m_exisitingContentEntries is populated accordingly which is required for serializing/ deserializing the cache entries.
The unit test iterates three times each times trying to load/reload the cache. Since the unit test is quick the garbageCollector is forced to abruptly terminate its operation before it can populate the map accordingly.
Once we hit closeAsync method. The CancellationToken is passed to garbageCollector forcing it to shut down its operations leaving the cache in an inconsistent state, but by awaiting the task we ensure that this does not happen.
By skipping the passing of CancellaitionToken to the gc we can avoid this race condition.
[Logs with Success and Failure.txt](https://dev.azure.com/mseng/9ed2c125-1cd5-4a17-886b-9d267f3a5fab/_apis/git/repositories/50d331c7-ea65-45eb-833f-0303c6c2387e/pullRequests/803195/attachments/Logs%20with%20Success%20and%20Failure.txt)
If we observe the logs we see that in the third iteration where we try to load the cache we do not see the garbage collector run at all(Can Collect print statement is missing in the failure run).
Related work items: #2191478
Use case-sensitive matching repo alias in pipeline definitions.
This is a workaround for bug #2210200
Without this, our pipeline run is neither tagged as 'Official' or 'Unofficial'; the tag is simply empty. This will fail our release pipeline.
Related work items: #2210200
The RunCheckinTests part of rolling build validation simulates some worker disconnect scenarions which generate warnings and an error. This change removed those expected errors from showing up and cluttering the Azure DevOps console.
If someone explicitly passes /fancyconsole in an AzureDevOps build, hyperlinks would be enabled. That's a weird configuration but some of the BuildXL repo automation was doing it. This additional check prevents that case.
Let users specify the total build timeout using `/buildTimeoutMins`. Before this PR the only way was via an engine env var (the CB runner uses this).
Deprecate /BuildWaitTimeout CLI. This flag (unrelated to the functionality being added here) is not being used and it has a confusing name given the new flag being added.
This pr is trying to address
1. Event dropping during the worker report execution log to orchestrator. The possible reasons are:
The execution log buffer is filled by another thread while we report it.
NotifyStream is deactivated before BinaryLogger completes all write events
2. CompareEvent trys to compare RecordFileForBuildManifest events
Changes in this pr:
1. Copy current content from stream and report to orchestrartor. Same as FlushManifestEvents
2. Track RecordFileForBuildManifest event in EventStatsExecutionLogTarget on worker.
3. Calculate total sent execution log size when reportExecutionLog succeed
4. Remove deactivate NotifyStream from Deactivate Execution Log. The stream will be disposed once the log target is disposed. Only deactivate it when we want to early cancel the sending action
5. Log Pending events into stats for manifest events and execution log reporting
Related work items: #2160034
Use TwoLevelCache to implement a bxl-side cache factory that allows to create a local cache + plain remote blob one. Existing knobs for both caches are preserved.
Previous change addressed the crash when loading cache after bxl cancelled. Crash still happen if load the cache after scheduler been cancelled.
To avoid crash, add access to scheduler cancellation token in PipTwoPhaseCache and check whether cancellation been requested before load cache.
Related work items: #2199303
Windows Terminal and other terminals support clickable hyperlinks. This change makes the link to the BuildXL log directory a clickable hyperlink to be opened by the operating system's file explorer of choice. This works on Windows and Linux.
I disabled it when run through an SSH session since the link would be a path on the host and the files may not exist on the SSH client. If run in a VSCode SSH session, VSCode has parsing for paths and does the right thing anyway.
This does change the behavior if you're running through a VSCode wrapped terminal locally. Previously, VSCode would attempt to show it through its file browser. Now it will use the file explorer of the OS instead. I didn't think this was a big deal and wasn't worthy of a config option.
Allow deduplicating content from ephemeral local CAS with CASaaS local content by always putting content into CASaaS first and then transferring hardlink to ephemeral local cas.
Flushing changes to the filesystem are intended for just-produced outputs. Do not attempt to do that for the source of a copy file pip when tracking a potential symlink chain. That may conflict with an attempt to hash a source file and throw a sharing violation.
Related work items: #2203343
When we the schedule hits an internal error, we aggressively terminate the build (DX0249). As a result, service pips (and their shutdown pips) will / might not exit cleanly. If they do, do not log `ScheduleServicePipFailed` / `ScheduleServicePipShuttingDownFailed` errors.
Related work items: #2199314
Allow users to specify reclassifications on operations resolved by the observed input processor. Users can configure rules in their specification that individual pips can opt into, where some observation types are reclassified after being processed by the observed input processor.
The rule matching happens both after execution and when processing pathsets for the pip on cache lookup. Future optimizations might avoid some of the processings on cache lookup.
Note that any changes in these rules results in a weak fingerprint change for the pips, as there is no way to know if observations for the pip have to be reclassified with the new rules.
To learn more about the motivations for this change, refer to [this document](https://microsoft.sharepoint.com/:w:/t/1ES2/EXOegdiLQkRNt5bE7nhFic8BJBYKcETrqcY-gI2_jTN9zw?e=UEIxAg)
Related work items: #2185890
* Fix cache generation condition logic so it is triggered by the right data
* Establish a precedence order between different cache generation options
* Get rid of IAdoBuildRunnerService (there are no intentions to have other implementations for the service)
Michael Pysson
Sahiti Chandramouli
Qi Wang
Serge Mera
BuildXL-Pipelines
Pasindu Gunasekara 🍣
Iman Narasamdya
Oleksii Kononenko
Lance Collins
Marcelo Lynch 🧉
Julian Bayardo
