Pipeline Generator has been updated to support generating CI triggers for public pipelines. This PR moves the version of pipeline generator we use to this new version. This will impact the scheduled pipeline generation job, as well as the single generation job and the prepare pipelines jobs.
From a functional standpoint, the generate matrix job only takes a few seconds to run. Depending on the repository, the job can take a total time of 4 minutes before the test jobs are able to start, because azure pipelines does a clone of the entire repository (among other things). This PR makes a change to skip the azure pipelines checkout step, and add a lightweight git clone along with a sparse checkout of just the `eng` and service directory locations (which contain the matrix scripts and/or matrix configs).
This change reduces the total job runtime to around 20 seconds from 4 minutes (repo checkouts + auto-injected policy steps).
When importing a matrix from another matrix, the intended behavior is that any duplicate parameter keys favor the value in the matrix doing the import. This PR fixes a bug where this did not happen (instead the imported matrix's value is favored).
* Small fix that validates for leaking implementation code if the parameter string is fully-qualified
* Don't include the maven parent or maven properties to avoid diff issues in apiview auto API workflow
- Remove BuildType parameter as we can default it from package properties
- Stop passing BuildType and GroupId and instead defaul them from package properties
- Enable StrictMode to help identify potential errors
- Start passing sdktype and isnewsdk properties to devops script
- Sync latest changes with devops work item to fix a couple bugs
Once https://github.com/Azure/azure-sdk-for-java/pull/19495 is merged and we merge this change the buildtype and groupid for java will come from the package properties.
This PR is a port of functionality that is currently duplicated across the net/java/python repositories. The intent was to settle on an implementation before moving it to the /eng/common/scripts directory. After merge to this location, I'll update the net/java/python and js repos to point to the `eng/common/scripts` location and remove the scripts from `eng/scripts`.
Here is the PR text used against the other repos for reference:
This adds scripts, docs and samples supporting dynamic, cross-product matrix generation for azure pipeline jobs.
It aims to replicate the [cross-product matrix functionality in github actions](https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#example-running-with-more-than-one-version-of-nodejs),
but also adds some additional features like sparse matrix generation, cross-product includes and excludes, parameter grouping and matrix filters.
This functionality is made possible by the ability for the azure pipelines yaml to take a [dynamic variable as an input
for a job matrix definition](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#multi-job-configuration) (see the code sample at the bottom of the linked section).
See the README.md file for more details on the config file syntax and usage, as well as implementation details.
The tests (`test-matrix-functions.tests.ps1`) contain a lot of detail on expected data structures at various processing stages. The `-`test-matrix-functions.ps1` file could perhaps be split up or use some more organization, so let me know if it's hard to navigate.
Example:
```
{
"displayNames": {
"true": "TestFromSource"
},
"matrix": {
"Agent": {
"ubuntu-18.04": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" },
"windows-2019": { "OSVmImage": "MMS2019", "Pool": "azsdk-pool-mms-win-2019-general" },
"macOS-10.15": { "OSVmImage": "macOS-10.15", "Pool": "Azure Pipelines" }
},
"JavaTestVersion": [ "1.8", "1.11" ],
"AZURE_TEST_HTTP_CLIENTS": [ "okhttp", "netty" ]
},
"include": [
{
"Agent": {
"ubuntu-18.04": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" }
},
"JavaTestVersion": "1.11",
"AZURE_TEST_HTTP_CLIENTS": "netty",
"TestFromSource": true
}
]
}
```
Sparse matrix job generation in a pipeline: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=705622&view=results
Related discussion: https://github.com/microsoft/azure-pipelines-yaml/issues/20
* Create package property file using artifact name
* Add option to pass config file dir
* Set default config file directory to avoid script execution failure
This PR reworks the verify agent OS logic so that the agent pool and image are passed into a PowerShell script, and then is checked against a list of valid configuration entries.
praveenkuttappan
Heath Stewart
Mitch Denny
Ben Broderick Phillips
Chidozie Ononiwu
Wes Haggard
Jonathan Giles
Christopher Scott
Minghao Chen
Daniel Jurek
Sima Zhu
Srikanta
