This PR fixes the discrepancy between the artifact services package that requires 4.4.0 version and BuildXL.Utilities that used to require 4.3.0 version. Without this change referencing BUildXL.Utilities package was causing a nuget package downgrade.
Related work items: #2039361
Add a --release option to bxl.sh. This works as an orthogonal configuration option for all the existing build options (e.g. ' --minimal --release' , or '--deploy-dev --release'. Default is still debug.
The pattern "*.*" in directory enumeration means include all members.
Our observed input processor turns it into a regex pattern that exclude any member whose name has no '.'. This means we miss all file and directory members that don't have any extension in our fingerprint.
You can run `dir *.*` in your command prompt to test it. But if you run that in Powershell terminal, all names without extension are filtered out. But in this case, for Windows, we follow the semantics of `FindFirstFile` and `PathMatchSpec`.
Apparently some tools like `python` performs directory enumerations with this legacy pattern.
Our public rolling validation tries to scrub this directory but fails with a warning because it is not a scrubbable mount. The result is we include unwanted packages in the produced artifact
The Limiting factors summary at the end of the build had a heuristic for ConcurrencyLimit. That was calculated based on there being free resources on the machine and more work available to be run. Evolutions to the scheduler eventually codified this heuristic into the UnavailableSlots counter. This superseded the ConcurrencyLimit in precedence in ExecutionSampler. So ConcurrentyLimit was basically always getting logged at 0%.
I consolidated the two keeping the original ConcurrencyLimit terminology since that was used in the performance summary at the end of the build log.
Related work items: #2033907
Adds a specific error message to recommend people connect to VPN to access the VSTS cache. Also categorize this specific cache init failure as a user error.
Reporting process creation for fork/clone on the parent has the problem that report lines from the child may non-deterministically arrive before the process creation report line. This causes that on managed side we don't have a process entry to associate those reports to, and therefore an allowlist that needs the current executable for a given access cannot be applied. This PR moves the reporting to the child.
Related work items: #2038280
The debugging documentation currently doesn't list the namespace of the functions it describes, causing developers to spend time reverse engineering the debug object.
This commit adds the namespace to the documentation to make it clear to developers that these functions need to be prefixed by the Debug namespace when invoking them.
BuildXL currently snaps total machine memory (working set) at the beginning of the build. This is normally a safe assumption, but builds on machines that are VMs with dynamic memory on the host will vary the amount of assigned memory while the build happens. This change allows what is used for modeling resources to grow while the build is running.
This PR adds `PolySharpAttributes` folder into BuildXLSdk to clearly separate which attributes the SDK adds in order to use the latest C# language feature with an ability to target the older runtimes.
The PR also removes `isDotNetCoreApp` property from `BuildXLSdk` in favor of `isDotNetCore` property.
The PR also adds a few missing attributes, like `ModuleInitializerAttribute`, `StringSyntaxAttribute`, `SkipLocalInitAttribute` and also moves `StackTraceHiddenAttribute` from cache interfaces project into `PolySharpAttributes` folder.
Enable dev logs across the board (even for dev builds): when not explicitly specified otherwise, a dev log will be generated by cherry-picking relevant events (same events used today in CB)
Related work items: #2038279
This overload is not used in the Linux sandbox, and we can simplify the code by removing it and slightly changing the MacOS code that was using it
Related work items: #2037865
The tools might pass `NULL` as arguments to the functions we're interposing. We were passing around the `char*`s happily without taking this into account, were very close to null pointer dereferencing (on automatic `std::string` conversion) in some cases, and also trying to create reports with empty paths in other cases.
This PR removes the `` create_access(const char *syscallName, es_event_type_t eventType, const std::string &reportPath, const std::string &secondPath ...);`` signature in favor of one using `char*` to avoid this pitfall, and adds some report skipping when the paths involved are `NULL`
Related work items: #2036506, #2036745
Turns out if you're dealing with content that is 2GB or larger, you can fail to put the content into the cache because of this overflow. I couldn't find any reason why this cast was being made, so I'll just assume it was an oversight during development.
We are not using the explicitly reported accesses for anything (we only consume the unexpected ones). Test coverage already in place for graph cache miss/hit. The visible effect is that we don't log detour messages for the Linux sandbox on graph construction.
PerformanceCollector has a dependency on Win32, and SandboxedProcess.cs has a dependency on PerformanceCollector. However, the only thing that's needed is the Aggregation Class which I've broken out of PerformanceCollector and into BXL.Utilities.Core.
The environment might be cleared at any point by the running process, making us lose the values if we're relying on an environment variable being set for it. Instead, store the path in the observer for subsequent uses.
The environment being cleared is not a theoretical problem, it was observed in practice, and it made us crash, see: #2036917
Related work items: #2036917
Directories created by the build need to take into consideration directories deleted by the build. This can only happen on execution (not on cache replay), but in that case a directory that was originally deleted by the build should still be considered a source file (in the same way as rewritten files)
Related work items: #2037034
This PR removes the concept of cache sessions that can only be read from. This:
1. Isn't actually used in practice by anyone, and it is a hinderance when changing the code.
2. Is actually a lie, because the cache doesn't truly support being read-only. Read-only operations can and do trigger mutations inside of the cache.
All cache sessions are now writeable.
BXL.Utilties.Collections depends on BXL.Cache.ContentStore.Hashing, but only 2 files actually use anything from it. Bloomfilter which doesn't appear to be used by anything and MurmurHashEngine which is only used by Storage so I moved it there
This PR adds an option to use span-based serialization/deserialization logic for LLS for code unification and performance reasons.
The PR adds a flag to switch from the old to the new logic and the test that shows the difference.
The new version is 3-x faster than the old version.
More specific data:
Old version: WriteDuration: 9444ms, ReadDuration: 9696ms
New version: WriteDuration: 3831ms, ReadDuration: 3119ms
Related work items: #2014041
Breaking Instrumentation.Common Dependency from BXL..Utilities.Core as it pulls in aria package. And the goal of BXL.Utilities.Core is to depend on no external packages
The original barrier to setting this on-by-default was the zero-byte copy bug, which has since been fixed.
This change always performs the directory handle check, then falls back to normal copy behavior if unsupported or a failure occures.
As an escape hatch, environment variable `DisableCopyOnWriteWin` will allow forcing normal copies.
Refactored some code around BXL.Utilities.Tasks to break that dependency from Sandboxing related code. Also moved some files that Sandboxing code depended on from Utilities to Utilities.Core as a part of this change.
CloudDev Build Account
Sergey Tepliakov
Serge Mera
Iman Narasamdya
Michael Pysson
Julian Bayardo
Semih Okur
Will Li
Brian Barthel
Marcelo Lynch 🧉
Juan Carlos Guzman Islas
Sahiti Chandramouli
Christian Castaneda