The variables persist beyond the scope of the single task. This is useful to run RunBxlWithPAT in a pre-build task and then have the build task be exposed to these variables.
This change looks like a net negative on the surface (why replace a single script with two separate pieces?) but we need it to migrate the pipelines to 1ESPT using the BuildXL workflow, which will call BuildXL outside of any wrapping script
- Adds two new npmrc files:
- .internal-npmrc can be used with cloudbuild or vsts-npm-auth to generate a token locally
- .ci-npmrc is used for ADO builds using an environment variable to store the token
- vsts-npm-auth does not support non-Windows platforms, so we need to manually do what it does on bxl.sh
- TODO: once merged, enable npm authentication on CloudBuild for bxl PR and rolling queues.
Related work items: #2115724
Changes need for developer experience for BuildXL-selfhost remoting:
* No need to install AnyBuild
* Use AnyBuild.json for repo configuration
* Simplify bxl.ps1 options
* Remove redundant engine environment settings
**TODO**
- [X] Use prod cluster
- [ ] Selectively include test pips (e.g., cache tests) to can-be-remoted pips
- [ ] Remove `/server-`
- [ ] Enable shared compilation
More pips will be added once it's proven successfully in the pipeline.
/server- currently cannot be removed (see details in bxl.ps1).
shared compilation still needs more investigation.
Related work items: #1915141
Currently it only remotes process pip with "compile" tag.
The remoting configuration is still not optimal currently, particularly for the beefy pool we have (32 cores).
In this run, https://dev.azure.com/mseng/Domino/_build/results?buildId=16779538&view=results, with remoting multiplier 1.5, and num of leases is assumed to be 2*maxproc, there are 42 processes going to AnyBuild, but 22 of them failed to find agents because AnyBuild is over saturated.
In this run, https://dev.azure.com/mseng/Domino/_build/results?buildId=16779745&view=results, with remoting multiplier 1.25 and num of leases 24, there are only 17 processes going to AnyBuild, and all of them successfully execute remotely.
More work is still needed for optimal configuration, e.g.,
- Feedback from AnyBuild about available leases, etc.
Related work items: #1912875
Marcelo Lynch 🧉
Pasindu Gunasekara 🍣
Iman Narasamdya