Move supporting matrix generator assets out of `eng/common` (#5027)
This PR contributes to addressing: - #4995 by moving out of `eng/common/scripts/job-matrix` assets that should not be mirrored to language repos. These assets include documentation, samples, and tests. ## Changes made - Moved contents of `eng/common/scripts/job-matrix/README.md` to newly introduced `doc/common/matrix_generator.md`; - Replaced contents of `eng/common/scripts/job-matrix/README.md` with a pointer to `doc/common/matrix_generator.md`; - Moved contents of `eng/common/scripts/job-matrix/scripts` and `eng/common/scripts/job-matrix/tests` to corresponding directories in newly introduced directory of `eng/common-tests/matrix-generator`. Note the introduced directory and doc file use `matrix generator` as name, not `job matrix`. This matches the terminology used in the documentation. ## Changes NOT made - The contents of files moved from `scripts` and `tests` directories have not been updated at all. The files have been moved unchanged. - The contents of the moved `README.md` have not been updated, even though they should, as they refer to paths that have changed as a result of the move. To keep the PR diff clean, I will instead update them in a separate PR: - #5023
This commit is contained in:
Konrad Jamrozik
GitHub
Родитель
f114ff8825
Коммит
ee399734c0
|
|
@ -0,0 +1,561 @@
|
|||
# Azure Pipelines Matrix Generator
|
||||
|
||||
* [Usage in a pipeline](#usage-in-a-pipeline)
|
||||
* [Matrix config file syntax](#matrix-config-file-syntax)
|
||||
* [Fields](#fields)
|
||||
* [matrix](#matrix)
|
||||
* [include](#include)
|
||||
* [exclude](#exclude)
|
||||
* [displayNames](#displaynames)
|
||||
* [$IMPORT](#import)
|
||||
* [Matrix Generation behavior](#matrix-generation-behavior)
|
||||
* [all](#all)
|
||||
* [sparse](#sparse)
|
||||
* [include/exclude](#includeexclude)
|
||||
* [displayNames](#displaynames-1)
|
||||
* [Filters](#filters)
|
||||
* [Replace/Modify/Append](#replacemodifyappend-values)
|
||||
* [NonSparseParameters](#nonsparseparameters)
|
||||
* [Under the hood](#under-the-hood)
|
||||
* [Testing](#testing)
|
||||
|
||||
|
||||
This directory contains scripts supporting dynamic, cross-product matrix generation for azure pipeline jobs.
|
||||
It aims to replicate the [cross-product matrix functionality in github actions](https://docs.github.com/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#example-running-with-more-than-one-version-of-nodejs),
|
||||
but also adds some additional features like sparse matrix generation, cross-product includes and excludes, and programmable matrix filters.
|
||||
|
||||
This functionality is made possible by the ability for the azure pipelines yaml to take a [dynamic variable as an input
|
||||
for a job matrix definition](https://docs.microsoft.com/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#multi-job-configuration) (see the code sample at the bottom of the linked section).
|
||||
|
||||
## Usage in a pipeline
|
||||
|
||||
In order to use these scripts in a pipeline, you must provide a config file and call the matrix creation script within a powershell job.
|
||||
|
||||
For a single matrix, you can include the `/eng/common/pipelines/templates/jobs/archetype-sdk-tests-generate.yml` template in a pipeline (see /eng/common-tests/matrix-generator/samples/matrix-test.yml for a full working example):
|
||||
|
||||
```
|
||||
jobs:
|
||||
- template: /eng/common/pipelines/templates/jobs/archetype-sdk-tests-generate.yml
|
||||
parameters:
|
||||
MatrixConfigs:
|
||||
- Name: base_product_matrix
|
||||
Path: eng/common-tests/matrix-generator/samples/matrix.json
|
||||
Selection: all
|
||||
NonSparseParameters:
|
||||
- framework
|
||||
GenerateVMJobs: true
|
||||
- Name: sparse_product_matrix
|
||||
Path: eng/common-tests/matrix-generator/samples/matrix.json
|
||||
Selection: sparse
|
||||
GenerateVMJobs: true
|
||||
JobTemplatePath: /eng/common-tests/matrix-generator/samples/matrix-job-sample.yml
|
||||
AdditionalParameters: []
|
||||
CloudConfig:
|
||||
SubscriptionConfiguration: $(sub-config-azure-cloud-test-resources)
|
||||
Location: eastus2
|
||||
Cloud: Public
|
||||
MatrixFilters: []
|
||||
MatrixReplace: []
|
||||
PreGenerationSteps: []
|
||||
```
|
||||
|
||||
### A note regarding PreGenerationSteps
|
||||
|
||||
The generation template laid out above runs as its own job. A limitation of this method is that it disallows any runtime matrix customization due to the fact that an individual job clones the targeted build SHA. The stepList `PreGenerationSteps` allows users to update matrix json however they like prior to actually invoking the matrix generation. Injected steps are run after the repository checkout, but before any matrix generation is invoked.
|
||||
|
||||
## Matrix config file syntax
|
||||
|
||||
Matrix parameters can either be a list of strings, or a set of grouped strings (represented as a hash). The latter parameter
|
||||
type is useful for when 2 or more parameters need to be grouped together, but without generating more than one matrix permutation.
|
||||
|
||||
```
|
||||
"matrix": {
|
||||
"<parameter1 name>": [ <values...> ],
|
||||
"<parameter2 name>": [ <values...> ],
|
||||
"<parameter set>": {
|
||||
"<parameter set 1 name>": {
|
||||
"<parameter set 1 value 1": "value",
|
||||
"<parameter set 1 value 2": "<value>",
|
||||
},
|
||||
"<parameter set 2 name>": {
|
||||
"<parameter set 2 value 1": "value",
|
||||
"<parameter set 2 value 2": "<value>",
|
||||
}
|
||||
}
|
||||
}
|
||||
"include": [ <matrix>, <matrix>, ... ],
|
||||
"exclude": [ <matrix>, <matrix>, ... ],
|
||||
"displayNames": { <parameter value>: <human readable override> }
|
||||
```
|
||||
|
||||
See `samples/matrix.json` for a full sample.
|
||||
|
||||
### Fields
|
||||
|
||||
#### matrix
|
||||
|
||||
The `matrix` field defines the base cross-product matrix. The generated matrix can be full or sparse.
|
||||
|
||||
Example:
|
||||
```
|
||||
"matrix": {
|
||||
"operatingSystem": [
|
||||
"windows-2022",
|
||||
"ubuntu-18.04",
|
||||
"macos-11"
|
||||
],
|
||||
"framework": [
|
||||
"net461",
|
||||
"netcoreapp2.1",
|
||||
"net50"
|
||||
],
|
||||
"additionalTestArguments": [
|
||||
"",
|
||||
"/p:UseProjectReferenceToAzureClients=true",
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### include
|
||||
|
||||
The `include` field defines any number of matrices to be appended to the base matrix after processing exclusions.
|
||||
|
||||
```
|
||||
# matrix entry format:
|
||||
{
|
||||
"a": 1,
|
||||
"b": 2,
|
||||
"c": 3,
|
||||
}
|
||||
|
||||
# Include field in a matrix config
|
||||
{
|
||||
"include": [
|
||||
{
|
||||
"a": 1,
|
||||
"b": 2
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
#### exclude
|
||||
|
||||
The `exclude` field defines any number of matrices to be removed from the base matrix. Exclude parameters can be a partial
|
||||
set, meaning as long as all exclude parameters match against a matrix entry (even if the matrix entry has additional parameters),
|
||||
then it will be excluded from the matrix. For example, the below entry will match the exclusion and be removed:
|
||||
|
||||
```
|
||||
# matrix entry format:
|
||||
{
|
||||
"a": 1,
|
||||
"b": 2,
|
||||
"c": 3,
|
||||
}
|
||||
|
||||
# Exclude field in a matrix config
|
||||
{
|
||||
"exclude": [
|
||||
{
|
||||
"a": 1,
|
||||
"b": 2
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### displayNames
|
||||
|
||||
Specify any overrides for the azure pipelines definition and UI that determines the matrix job name. If some parameter
|
||||
values are too long or unreadable for this purpose (e.g. a command line argument), then you can replace them with a more
|
||||
readable value here. For example:
|
||||
|
||||
```
|
||||
"displayNames": {
|
||||
"/p:UseProjectReferenceToAzureClients=true": "UseProjectRef"
|
||||
},
|
||||
"matrix": {
|
||||
"additionalTestArguments": [
|
||||
"/p:UseProjectReferenceToAzureClients=true"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### $IMPORT
|
||||
|
||||
Matrix configs can also import another matrix config. The effect of this is the imported matrix will be generated,
|
||||
and then the importing config will be combined with that matrix (as if each entry of the imported matrix was a parameter).
|
||||
To import a matrix, add a parameter with the key `$IMPORT`:
|
||||
|
||||
```
|
||||
"matrix": {
|
||||
"$IMPORT": "path/to/matrix.json",
|
||||
"JavaVersion": [ "1.8", "1.11" ]
|
||||
}
|
||||
```
|
||||
|
||||
Importing can be useful, for example, in cases where there is a shared base matrix, but there is a need to run it
|
||||
once for each instance of a language version. Importing does not support overriding duplicate parameters. To achieve
|
||||
this, use the [Replace](#replacemodifyappend-values) argument instead.
|
||||
|
||||
The `Selection` and `NonSparseParameters` parameters are respected when generating an imported matrix.
|
||||
|
||||
The processing order is as follows:
|
||||
|
||||
Given a matrix and import matrix like below:
|
||||
```
|
||||
{
|
||||
"matrix": {
|
||||
"$IMPORT": "example-matrix.json",
|
||||
"endpointType": [ "storage", "cosmos" ],
|
||||
"JavaVersion": [ "1.8", "1.11" ]
|
||||
},
|
||||
"include": [
|
||||
{
|
||||
"operatingSystem": "windows",
|
||||
"mode": "TestFromSource",
|
||||
"JavaVersion": "1.8"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
### example-matrix.json to import
|
||||
{
|
||||
"matrix": {
|
||||
"operatingSystem": [ "windows", "linux" ],
|
||||
"client": [ "netty", "okhttp" ]
|
||||
},
|
||||
"include": [
|
||||
{
|
||||
"operatingSystem": "mac",
|
||||
"client": "netty"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
1. The base matrix is generated (sparse in this example):
|
||||
```
|
||||
{
|
||||
"storage_18": {
|
||||
"endpointType": "storage",
|
||||
"JavaVersion": "1.8"
|
||||
},
|
||||
"cosmos_111": {
|
||||
"endpointType": "cosmos",
|
||||
"JavaVersion": "1.11"
|
||||
}
|
||||
}
|
||||
```
|
||||
1. The imported base matrix is generated (sparse in this example):
|
||||
```
|
||||
{
|
||||
"windows_netty": {
|
||||
"operatingSystem": "windows",
|
||||
"client": "netty"
|
||||
},
|
||||
"linux_okhttp": {
|
||||
"operatingSystem": "linux",
|
||||
"client": "okhttp"
|
||||
}
|
||||
}
|
||||
```
|
||||
1. Includes/excludes from the imported matrix get applied to the imported matrix
|
||||
```
|
||||
{
|
||||
"windows_netty": {
|
||||
"operatingSystem": "windows",
|
||||
"client": "netty"
|
||||
},
|
||||
"linux_okhttp": {
|
||||
"operatingSystem": "linux",
|
||||
"client": "okhttp"
|
||||
},
|
||||
"mac_netty": {
|
||||
"operatingSystem": "mac",
|
||||
"client": "netty"
|
||||
}
|
||||
}
|
||||
```
|
||||
1. The base matrix is multipled by the imported matrix (in this case, the base matrix has 2 elements, and the imported
|
||||
matrix has 3 elements, so the product is a matrix with 6 elements:
|
||||
```
|
||||
"storage_18_windows_netty": {
|
||||
"endpointType": "storage",
|
||||
"JavaVersion": "1.8",
|
||||
"operatingSystem": "windows",
|
||||
"client": "netty"
|
||||
},
|
||||
"storage_18_linux_okhttp": {
|
||||
"endpointType": "storage",
|
||||
"JavaVersion": "1.8",
|
||||
"operatingSystem": "linux",
|
||||
"client": "okhttp"
|
||||
},
|
||||
"storage_18_mac_netty": {
|
||||
"endpointType": "storage",
|
||||
"JavaVersion": "1.8",
|
||||
"operatingSystem": "mac",
|
||||
"client": "netty"
|
||||
},
|
||||
"cosmos_111_windows_netty": {
|
||||
"endpointType": "cosmos",
|
||||
"JavaVersion": "1.11",
|
||||
"operatingSystem": "windows",
|
||||
"client": "netty"
|
||||
},
|
||||
"cosmos_111_linux_okhttp": {
|
||||
"endpointType": "cosmos",
|
||||
"JavaVersion": "1.11",
|
||||
"operatingSystem": "linux",
|
||||
"client": "okhttp"
|
||||
},
|
||||
"cosmos_111_mac_netty": {
|
||||
"endpointType": "cosmos",
|
||||
"JavaVersion": "1.11",
|
||||
"operatingSystem": "mac",
|
||||
"client": "netty"
|
||||
}
|
||||
}
|
||||
```
|
||||
1. Includes/excludes from the top-level matrix get applied to the multiplied matrix, so the below element will be added
|
||||
to the above matrix, for an output matrix with 7 elements:
|
||||
```
|
||||
"windows_TestFromSource_18": {
|
||||
"operatingSystem": "windows",
|
||||
"mode": "TestFromSource",
|
||||
"JavaVersion": "1.8"
|
||||
}
|
||||
```
|
||||
|
||||
## Matrix Generation behavior
|
||||
|
||||
#### all
|
||||
|
||||
`all` will output the full matrix, i.e. every possible permutation of all parameters given (p1.Length * p2.Length * ...).
|
||||
|
||||
#### sparse
|
||||
|
||||
`sparse` outputs the minimum number of parameter combinations while ensuring that all parameter values are present in at least one matrix job.
|
||||
Effectively this means the total length of a sparse matrix will be equal to the largest matrix dimension, i.e. `max(p1.Length, p2.Length, ...)`.
|
||||
|
||||
To build a sparse matrix, a full matrix is generated, and then walked diagonally N times where N is the largest matrix dimension.
|
||||
This pattern works for any N-dimensional matrix, via an incrementing index (n, n, n, ...), (n+1, n+1, n+1, ...), etc.
|
||||
Index lookups against matrix dimensions are calculated modulus the dimension size, so a two-dimensional matrix of 4x2 might be walked like this:
|
||||
|
||||
```
|
||||
index: 0, 0:
|
||||
o . . .
|
||||
. . . .
|
||||
|
||||
index: 1, 1:
|
||||
. . . .
|
||||
. o . .
|
||||
|
||||
index: 2, 2 (modded to 2, 0):
|
||||
. . o .
|
||||
. . . .
|
||||
|
||||
index: 3, 3 (modded to 3, 1):
|
||||
. . . .
|
||||
. . . o
|
||||
```
|
||||
|
||||
#### include/exclude
|
||||
|
||||
Include and exclude support additions and subtractions off the base matrix. Both include and exclude take an array of matrix values.
|
||||
Typically these values will be a single entry, but they also support the cross-product matrix definition syntax of the base matrix.
|
||||
|
||||
Include and exclude are parsed fully. So if a sparse matrix is called for, a sparse version of the base matrix will be generated, but
|
||||
the full matrix of both include and exclude will be processed.
|
||||
|
||||
Excludes are processed first, so includes can be used to add back any specific jobs to the matrix.
|
||||
|
||||
#### displayNames
|
||||
|
||||
In the matrix job output that azure pipelines consumes, the format is a dictionary of dictionaries. For example:
|
||||
|
||||
```
|
||||
{
|
||||
"net461_macOS1015": {
|
||||
"framework": "net461",
|
||||
"operatingSystem": "macos-11"
|
||||
},
|
||||
"net50_ubuntu1804": {
|
||||
"framework": "net50",
|
||||
"operatingSystem": "ubuntu-18.04"
|
||||
},
|
||||
"netcoreapp21_windows2022": {
|
||||
"framework": "netcoreapp2.1",
|
||||
"operatingSystem": "windows-2022"
|
||||
},
|
||||
"UseProjectRef_net461_windows2022": {
|
||||
"additionalTestArguments": "/p:UseProjectReferenceToAzureClients=true",
|
||||
"framework": "net461",
|
||||
"operatingSystem": "windows-2022"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The top level keys are used as job names, meaning they get displayed in the azure pipelines UI when running the pipeline.
|
||||
|
||||
The logic for generating display names works like this:
|
||||
|
||||
- Join parameter values by "_"
|
||||
a. If the parameter value exists as a key in `displayNames` in the matrix config, replace it with that value.
|
||||
b. For each name value, strip all non-alphanumeric characters (excluding "_").
|
||||
c. If the name is greater than 100 characters, truncate it.
|
||||
|
||||
#### Filters
|
||||
|
||||
Filters can be passed to the matrix as an array of strings, each matching the format of `<key>=<regex>`. When a matrix entry
|
||||
does not contain the specified key, it will default to a value of empty string for regex parsing. This can be used to specify
|
||||
filters for keys that don't exist or keys that optionally exist and match a regex, as seen in the below example.
|
||||
|
||||
Display name filters can also be passed as a single regex string that runs against the [generated display name](#displaynames) of the matrix job.
|
||||
The intent of display name filters is to be defined primarily as a top level variable at template queue time in the azure pipelines UI.
|
||||
|
||||
For example, the below command will filter for matrix entries with "windows" in the job display name, no matrix variable
|
||||
named "ExcludedKey", a framework variable containing either "461" or "5.0", and an optional key "SupportedClouds" that, if exists, must contain "Public":
|
||||
|
||||
```
|
||||
./Create-JobMatrix.ps1 `
|
||||
-ConfigPath samples/matrix.json `
|
||||
-Selection all `
|
||||
-DisplayNameFilter ".*windows.*" `
|
||||
-Filters @("ExcludedKey=^$", "framework=(461|5\.0)", "SupportedClouds=^$|.*Public.*")
|
||||
```
|
||||
|
||||
#### Replace/Modify/Append Values
|
||||
|
||||
Replacements for values can be passed to the matrix as an array of strings, each matching the format of `<keyRegex>=<valueRegex>/<replacementValue>`.
|
||||
The replace argument will find any permutations where the key fully matches the key regex and the value fully matches the value regex, and replace the value with
|
||||
the replacement specified.
|
||||
|
||||
NOTE:
|
||||
- The replacement value supports regex capture groups, enabling substring transformations, e.g. `Foo=(.*)-replaceMe/$1-replaced`. See the below examples for usage.
|
||||
- For each key/value, the first replacement provided that matches will be the only one applied.
|
||||
- If `=` or `/` characters need to be part of the regex or replacement, escape them with `\`.
|
||||
|
||||
For example, given a matrix config like below:
|
||||
|
||||
```
|
||||
{
|
||||
"matrix": {
|
||||
"Agent": {
|
||||
"ubuntu-1804": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" }
|
||||
},
|
||||
"JavaTestVersion": [ "1.8", "1.11" ]
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
The normal matrix output (without replacements), looks like:
|
||||
|
||||
```
|
||||
$ ./Create-JobMatrix.ps1 -ConfigPath <test> -Selection all
|
||||
{
|
||||
"ubuntu1804_18": {
|
||||
"OSVmImage": "MMSUbuntu18.04",
|
||||
"Pool": "azsdk-pool-mms-ubuntu-1804-general",
|
||||
"JavaTestVersion": "1.8"
|
||||
},
|
||||
"ubuntu1804_111": {
|
||||
"OSVmImage": "MMSUbuntu18.04",
|
||||
"Pool": "azsdk-pool-mms-ubuntu-1804-general",
|
||||
"JavaTestVersion": "1.11"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Passing in multiple replacements, the output will look like below. Note that replacing key/values that appear nested within a grouping
|
||||
will not affect that segment of the job name, since the job takes the grouping name (in this case "ubuntu1804").
|
||||
|
||||
The below example includes samples of regex grouping references, and wildcard key/value regexes:
|
||||
|
||||
```
|
||||
$ $replacements = @('.*Version=1.11/2.0', 'Pool=(.*ubuntu.*)-general/$1-custom')
|
||||
$ ../Create-JobMatrix.ps1 -ConfigPath ./test.Json -Selection all -Replace $replacements
|
||||
{
|
||||
"ubuntu1804_18": {
|
||||
"OSVmImage": "MMSUbuntu18.04",
|
||||
"Pool": "azsdk-pool-mms-ubuntu-1804-custom",
|
||||
"JavaTestVersion": "1.8"
|
||||
},
|
||||
"ubuntu1804_20": {
|
||||
"OSVmImage": "MMSUbuntu18.04",
|
||||
"Pool": "azsdk-pool-mms-ubuntu-1804-custom",
|
||||
"JavaTestVersion": "2.0"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### NonSparseParameters
|
||||
|
||||
Sometimes it may be necessary to generate a sparse matrix, but keep the full combination of a few parameters. The
|
||||
NonSparseParameters argument allows for more fine-grained control of matrix generation. For example:
|
||||
|
||||
```
|
||||
./Create-JobMatrix.ps1 `
|
||||
-ConfigPath /path/to/matrix.json `
|
||||
-Selection sparse `
|
||||
-NonSparseParameters @("JavaTestVersion")
|
||||
```
|
||||
|
||||
Given a matrix like below with `JavaTestVersion` marked as a non-sparse parameter:
|
||||
|
||||
```
|
||||
{
|
||||
"matrix": {
|
||||
"Agent": {
|
||||
"windows-2022": { "OSVmImage": "MMS2022", "Pool": "azsdk-pool-mms-win-2022-general" },
|
||||
"ubuntu-1804": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" },
|
||||
"macos-11": { "OSVmImage": "macos-11", "Pool": "Azure Pipelines" }
|
||||
},
|
||||
"JavaTestVersion": [ "1.8", "1.11" ],
|
||||
"AZURE_TEST_HTTP_CLIENTS": "netty",
|
||||
"ArmTemplateParameters": [ "@{endpointType='storage'}", "@{endpointType='cosmos'}" ]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
A matrix with 6 entries will be generated: A sparse matrix of Agent, AZURE_TEST_HTTP_CLIENTS and ArmTemplateParameters
|
||||
(3 total entries) will be multipled by the two `JavaTestVersion` parameters `1.8` and `1.11`.
|
||||
|
||||
NOTE: NonSparseParameters are also applied when generating an imported matrix.
|
||||
|
||||
#### Under the hood
|
||||
|
||||
The script generates an N-dimensional matrix with dimensions equal to the parameter array lengths. For example,
|
||||
the below config would generate a 2x2x1x1x1 matrix (five-dimensional):
|
||||
|
||||
```
|
||||
"matrix": {
|
||||
"framework": [ "net461", "netcoreapp2.1" ],
|
||||
"additionalTestArguments": [ "", "/p:SuperTest=true" ]
|
||||
"pool": [ "ubuntu-18.04" ],
|
||||
"container": [ "ubuntu-18.04" ],
|
||||
"testMode": [ "Record" ]
|
||||
}
|
||||
```
|
||||
|
||||
The matrix is stored as a one-dimensional array, with a row-major indexing scheme (e.g. `(2, 1, 0, 1, 0)`).
|
||||
|
||||
## Testing
|
||||
|
||||
The matrix functions can be tested using [pester](https://pester.dev/). The test command must be run from within the tests directory.
|
||||
|
||||
```
|
||||
$ cd <azure sdk tools repo root>/eng/common-tests/matrix-generator/tests
|
||||
$ Invoke-Pester
|
||||
|
||||
Starting discovery in 3 files.
|
||||
Discovery finished in 75ms.
|
||||
[+] /home/ben/sdk/azure-sdk-tools/eng/common-tests/matrix-generator/tests/job-matrix-functions.filter.tests.ps1 750ms (309ms|428ms)
|
||||
[+] /home/ben/sdk/azure-sdk-tools/eng/common-tests/matrix-generator/tests/job-matrix-functions.modification.tests.ps1 867ms (250ms|608ms)
|
||||
[+] /home/ben/sdk/azure-sdk-tools/eng/common-tests/matrix-generator/tests/job-matrix-functions.tests.ps1 2.71s (725ms|1.93s)
|
||||
Tests completed in 4.33s
|
||||
Tests Passed: 141, Failed: 0, Skipped: 4 NotRun: 0
|
||||
```
|
||||
|
|
@ -1,561 +1,5 @@
|
|||
# Azure Pipelines Matrix Generator
|
||||
|
||||
* [Usage in a pipeline](#usage-in-a-pipeline)
|
||||
* [Matrix config file syntax](#matrix-config-file-syntax)
|
||||
* [Fields](#fields)
|
||||
* [matrix](#matrix)
|
||||
* [include](#include)
|
||||
* [exclude](#exclude)
|
||||
* [displayNames](#displaynames)
|
||||
* [$IMPORT](#import)
|
||||
* [Matrix Generation behavior](#matrix-generation-behavior)
|
||||
* [all](#all)
|
||||
* [sparse](#sparse)
|
||||
* [include/exclude](#includeexclude)
|
||||
* [displayNames](#displaynames-1)
|
||||
* [Filters](#filters)
|
||||
* [Replace/Modify/Append](#replacemodifyappend-values)
|
||||
* [NonSparseParameters](#nonsparseparameters)
|
||||
* [Under the hood](#under-the-hood)
|
||||
* [Testing](#testing)
|
||||
The documentation for Azure Pipelines Matrix Generator can be found at
|
||||
|
||||
|
||||
This directory contains scripts supporting dynamic, cross-product matrix generation for azure pipeline jobs.
|
||||
It aims to replicate the [cross-product matrix functionality in github actions](https://docs.github.com/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#example-running-with-more-than-one-version-of-nodejs),
|
||||
but also adds some additional features like sparse matrix generation, cross-product includes and excludes, and programmable matrix filters.
|
||||
|
||||
This functionality is made possible by the ability for the azure pipelines yaml to take a [dynamic variable as an input
|
||||
for a job matrix definition](https://docs.microsoft.com/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#multi-job-configuration) (see the code sample at the bottom of the linked section).
|
||||
|
||||
## Usage in a pipeline
|
||||
|
||||
In order to use these scripts in a pipeline, you must provide a config file and call the matrix creation script within a powershell job.
|
||||
|
||||
For a single matrix, you can include the `/eng/common/pipelines/templates/jobs/archetype-sdk-tests-generate.yml` template in a pipeline (see /eng/common/scripts/job-matrix/samples/matrix-test.yml for a full working example):
|
||||
|
||||
```
|
||||
jobs:
|
||||
- template: /eng/common/pipelines/templates/jobs/archetype-sdk-tests-generate.yml
|
||||
parameters:
|
||||
MatrixConfigs:
|
||||
- Name: base_product_matrix
|
||||
Path: eng/common/scripts/job-matrix/samples/matrix.json
|
||||
Selection: all
|
||||
NonSparseParameters:
|
||||
- framework
|
||||
GenerateVMJobs: true
|
||||
- Name: sparse_product_matrix
|
||||
Path: eng/common/scripts/job-matrix/samples/matrix.json
|
||||
Selection: sparse
|
||||
GenerateVMJobs: true
|
||||
JobTemplatePath: /eng/common/scripts/job-matrix/samples/matrix-job-sample.yml
|
||||
AdditionalParameters: []
|
||||
CloudConfig:
|
||||
SubscriptionConfiguration: $(sub-config-azure-cloud-test-resources)
|
||||
Location: eastus2
|
||||
Cloud: Public
|
||||
MatrixFilters: []
|
||||
MatrixReplace: []
|
||||
PreGenerationSteps: []
|
||||
```
|
||||
|
||||
### A note regarding PreGenerationSteps
|
||||
|
||||
The generation template laid out above runs as its own job. A limitation of this method is that it disallows any runtime matrix customization due to the fact that an individual job clones the targeted build SHA. The stepList `PreGenerationSteps` allows users to update matrix json however they like prior to actually invoking the matrix generation. Injected steps are run after the repository checkout, but before any matrix generation is invoked.
|
||||
|
||||
## Matrix config file syntax
|
||||
|
||||
Matrix parameters can either be a list of strings, or a set of grouped strings (represented as a hash). The latter parameter
|
||||
type is useful for when 2 or more parameters need to be grouped together, but without generating more than one matrix permutation.
|
||||
|
||||
```
|
||||
"matrix": {
|
||||
"<parameter1 name>": [ <values...> ],
|
||||
"<parameter2 name>": [ <values...> ],
|
||||
"<parameter set>": {
|
||||
"<parameter set 1 name>": {
|
||||
"<parameter set 1 value 1": "value",
|
||||
"<parameter set 1 value 2": "<value>",
|
||||
},
|
||||
"<parameter set 2 name>": {
|
||||
"<parameter set 2 value 1": "value",
|
||||
"<parameter set 2 value 2": "<value>",
|
||||
}
|
||||
}
|
||||
}
|
||||
"include": [ <matrix>, <matrix>, ... ],
|
||||
"exclude": [ <matrix>, <matrix>, ... ],
|
||||
"displayNames": { <parameter value>: <human readable override> }
|
||||
```
|
||||
|
||||
See `samples/matrix.json` for a full sample.
|
||||
|
||||
### Fields
|
||||
|
||||
#### matrix
|
||||
|
||||
The `matrix` field defines the base cross-product matrix. The generated matrix can be full or sparse.
|
||||
|
||||
Example:
|
||||
```
|
||||
"matrix": {
|
||||
"operatingSystem": [
|
||||
"windows-2022",
|
||||
"ubuntu-18.04",
|
||||
"macos-11"
|
||||
],
|
||||
"framework": [
|
||||
"net461",
|
||||
"netcoreapp2.1",
|
||||
"net50"
|
||||
],
|
||||
"additionalTestArguments": [
|
||||
"",
|
||||
"/p:UseProjectReferenceToAzureClients=true",
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### include
|
||||
|
||||
The `include` field defines any number of matrices to be appended to the base matrix after processing exclusions.
|
||||
|
||||
```
|
||||
# matrix entry format:
|
||||
{
|
||||
"a": 1,
|
||||
"b": 2,
|
||||
"c": 3,
|
||||
}
|
||||
|
||||
# Include field in a matrix config
|
||||
{
|
||||
"include": [
|
||||
{
|
||||
"a": 1,
|
||||
"b": 2
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
#### exclude
|
||||
|
||||
The `exclude` field defines any number of matrices to be removed from the base matrix. Exclude parameters can be a partial
|
||||
set, meaning as long as all exclude parameters match against a matrix entry (even if the matrix entry has additional parameters),
|
||||
then it will be excluded from the matrix. For example, the below entry will match the exclusion and be removed:
|
||||
|
||||
```
|
||||
# matrix entry format:
|
||||
{
|
||||
"a": 1,
|
||||
"b": 2,
|
||||
"c": 3,
|
||||
}
|
||||
|
||||
# Exclude field in a matrix config
|
||||
{
|
||||
"exclude": [
|
||||
{
|
||||
"a": 1,
|
||||
"b": 2
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### displayNames
|
||||
|
||||
Specify any overrides for the azure pipelines definition and UI that determines the matrix job name. If some parameter
|
||||
values are too long or unreadable for this purpose (e.g. a command line argument), then you can replace them with a more
|
||||
readable value here. For example:
|
||||
|
||||
```
|
||||
"displayNames": {
|
||||
"/p:UseProjectReferenceToAzureClients=true": "UseProjectRef"
|
||||
},
|
||||
"matrix": {
|
||||
"additionalTestArguments": [
|
||||
"/p:UseProjectReferenceToAzureClients=true"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### $IMPORT
|
||||
|
||||
Matrix configs can also import another matrix config. The effect of this is the imported matrix will be generated,
|
||||
and then the importing config will be combined with that matrix (as if each entry of the imported matrix was a parameter).
|
||||
To import a matrix, add a parameter with the key `$IMPORT`:
|
||||
|
||||
```
|
||||
"matrix": {
|
||||
"$IMPORT": "path/to/matrix.json",
|
||||
"JavaVersion": [ "1.8", "1.11" ]
|
||||
}
|
||||
```
|
||||
|
||||
Importing can be useful, for example, in cases where there is a shared base matrix, but there is a need to run it
|
||||
once for each instance of a language version. Importing does not support overriding duplicate parameters. To achieve
|
||||
this, use the [Replace](#replacemodifyappend-values) argument instead.
|
||||
|
||||
The `Selection` and `NonSparseParameters` parameters are respected when generating an imported matrix.
|
||||
|
||||
The processing order is as follows:
|
||||
|
||||
Given a matrix and import matrix like below:
|
||||
```
|
||||
{
|
||||
"matrix": {
|
||||
"$IMPORT": "example-matrix.json",
|
||||
"endpointType": [ "storage", "cosmos" ],
|
||||
"JavaVersion": [ "1.8", "1.11" ]
|
||||
},
|
||||
"include": [
|
||||
{
|
||||
"operatingSystem": "windows",
|
||||
"mode": "TestFromSource",
|
||||
"JavaVersion": "1.8"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
### example-matrix.json to import
|
||||
{
|
||||
"matrix": {
|
||||
"operatingSystem": [ "windows", "linux" ],
|
||||
"client": [ "netty", "okhttp" ]
|
||||
},
|
||||
"include": [
|
||||
{
|
||||
"operatingSystem": "mac",
|
||||
"client": "netty"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
1. The base matrix is generated (sparse in this example):
|
||||
```
|
||||
{
|
||||
"storage_18": {
|
||||
"endpointType": "storage",
|
||||
"JavaVersion": "1.8"
|
||||
},
|
||||
"cosmos_111": {
|
||||
"endpointType": "cosmos",
|
||||
"JavaVersion": "1.11"
|
||||
}
|
||||
}
|
||||
```
|
||||
1. The imported base matrix is generated (sparse in this example):
|
||||
```
|
||||
{
|
||||
"windows_netty": {
|
||||
"operatingSystem": "windows",
|
||||
"client": "netty"
|
||||
},
|
||||
"linux_okhttp": {
|
||||
"operatingSystem": "linux",
|
||||
"client": "okhttp"
|
||||
}
|
||||
}
|
||||
```
|
||||
1. Includes/excludes from the imported matrix get applied to the imported matrix
|
||||
```
|
||||
{
|
||||
"windows_netty": {
|
||||
"operatingSystem": "windows",
|
||||
"client": "netty"
|
||||
},
|
||||
"linux_okhttp": {
|
||||
"operatingSystem": "linux",
|
||||
"client": "okhttp"
|
||||
},
|
||||
"mac_netty": {
|
||||
"operatingSystem": "mac",
|
||||
"client": "netty"
|
||||
}
|
||||
}
|
||||
```
|
||||
1. The base matrix is multipled by the imported matrix (in this case, the base matrix has 2 elements, and the imported
|
||||
matrix has 3 elements, so the product is a matrix with 6 elements:
|
||||
```
|
||||
"storage_18_windows_netty": {
|
||||
"endpointType": "storage",
|
||||
"JavaVersion": "1.8",
|
||||
"operatingSystem": "windows",
|
||||
"client": "netty"
|
||||
},
|
||||
"storage_18_linux_okhttp": {
|
||||
"endpointType": "storage",
|
||||
"JavaVersion": "1.8",
|
||||
"operatingSystem": "linux",
|
||||
"client": "okhttp"
|
||||
},
|
||||
"storage_18_mac_netty": {
|
||||
"endpointType": "storage",
|
||||
"JavaVersion": "1.8",
|
||||
"operatingSystem": "mac",
|
||||
"client": "netty"
|
||||
},
|
||||
"cosmos_111_windows_netty": {
|
||||
"endpointType": "cosmos",
|
||||
"JavaVersion": "1.11",
|
||||
"operatingSystem": "windows",
|
||||
"client": "netty"
|
||||
},
|
||||
"cosmos_111_linux_okhttp": {
|
||||
"endpointType": "cosmos",
|
||||
"JavaVersion": "1.11",
|
||||
"operatingSystem": "linux",
|
||||
"client": "okhttp"
|
||||
},
|
||||
"cosmos_111_mac_netty": {
|
||||
"endpointType": "cosmos",
|
||||
"JavaVersion": "1.11",
|
||||
"operatingSystem": "mac",
|
||||
"client": "netty"
|
||||
}
|
||||
}
|
||||
```
|
||||
1. Includes/excludes from the top-level matrix get applied to the multiplied matrix, so the below element will be added
|
||||
to the above matrix, for an output matrix with 7 elements:
|
||||
```
|
||||
"windows_TestFromSource_18": {
|
||||
"operatingSystem": "windows",
|
||||
"mode": "TestFromSource",
|
||||
"JavaVersion": "1.8"
|
||||
}
|
||||
```
|
||||
|
||||
## Matrix Generation behavior
|
||||
|
||||
#### all
|
||||
|
||||
`all` will output the full matrix, i.e. every possible permutation of all parameters given (p1.Length * p2.Length * ...).
|
||||
|
||||
#### sparse
|
||||
|
||||
`sparse` outputs the minimum number of parameter combinations while ensuring that all parameter values are present in at least one matrix job.
|
||||
Effectively this means the total length of a sparse matrix will be equal to the largest matrix dimension, i.e. `max(p1.Length, p2.Length, ...)`.
|
||||
|
||||
To build a sparse matrix, a full matrix is generated, and then walked diagonally N times where N is the largest matrix dimension.
|
||||
This pattern works for any N-dimensional matrix, via an incrementing index (n, n, n, ...), (n+1, n+1, n+1, ...), etc.
|
||||
Index lookups against matrix dimensions are calculated modulus the dimension size, so a two-dimensional matrix of 4x2 might be walked like this:
|
||||
|
||||
```
|
||||
index: 0, 0:
|
||||
o . . .
|
||||
. . . .
|
||||
|
||||
index: 1, 1:
|
||||
. . . .
|
||||
. o . .
|
||||
|
||||
index: 2, 2 (modded to 2, 0):
|
||||
. . o .
|
||||
. . . .
|
||||
|
||||
index: 3, 3 (modded to 3, 1):
|
||||
. . . .
|
||||
. . . o
|
||||
```
|
||||
|
||||
#### include/exclude
|
||||
|
||||
Include and exclude support additions and subtractions off the base matrix. Both include and exclude take an array of matrix values.
|
||||
Typically these values will be a single entry, but they also support the cross-product matrix definition syntax of the base matrix.
|
||||
|
||||
Include and exclude are parsed fully. So if a sparse matrix is called for, a sparse version of the base matrix will be generated, but
|
||||
the full matrix of both include and exclude will be processed.
|
||||
|
||||
Excludes are processed first, so includes can be used to add back any specific jobs to the matrix.
|
||||
|
||||
#### displayNames
|
||||
|
||||
In the matrix job output that azure pipelines consumes, the format is a dictionary of dictionaries. For example:
|
||||
|
||||
```
|
||||
{
|
||||
"net461_macOS1015": {
|
||||
"framework": "net461",
|
||||
"operatingSystem": "macos-11"
|
||||
},
|
||||
"net50_ubuntu1804": {
|
||||
"framework": "net50",
|
||||
"operatingSystem": "ubuntu-18.04"
|
||||
},
|
||||
"netcoreapp21_windows2022": {
|
||||
"framework": "netcoreapp2.1",
|
||||
"operatingSystem": "windows-2022"
|
||||
},
|
||||
"UseProjectRef_net461_windows2022": {
|
||||
"additionalTestArguments": "/p:UseProjectReferenceToAzureClients=true",
|
||||
"framework": "net461",
|
||||
"operatingSystem": "windows-2022"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The top level keys are used as job names, meaning they get displayed in the azure pipelines UI when running the pipeline.
|
||||
|
||||
The logic for generating display names works like this:
|
||||
|
||||
- Join parameter values by "_"
|
||||
a. If the parameter value exists as a key in `displayNames` in the matrix config, replace it with that value.
|
||||
b. For each name value, strip all non-alphanumeric characters (excluding "_").
|
||||
c. If the name is greater than 100 characters, truncate it.
|
||||
|
||||
#### Filters
|
||||
|
||||
Filters can be passed to the matrix as an array of strings, each matching the format of `<key>=<regex>`. When a matrix entry
|
||||
does not contain the specified key, it will default to a value of empty string for regex parsing. This can be used to specify
|
||||
filters for keys that don't exist or keys that optionally exist and match a regex, as seen in the below example.
|
||||
|
||||
Display name filters can also be passed as a single regex string that runs against the [generated display name](#displaynames) of the matrix job.
|
||||
The intent of display name filters is to be defined primarily as a top level variable at template queue time in the azure pipelines UI.
|
||||
|
||||
For example, the below command will filter for matrix entries with "windows" in the job display name, no matrix variable
|
||||
named "ExcludedKey", a framework variable containing either "461" or "5.0", and an optional key "SupportedClouds" that, if exists, must contain "Public":
|
||||
|
||||
```
|
||||
./Create-JobMatrix.ps1 `
|
||||
-ConfigPath samples/matrix.json `
|
||||
-Selection all `
|
||||
-DisplayNameFilter ".*windows.*" `
|
||||
-Filters @("ExcludedKey=^$", "framework=(461|5\.0)", "SupportedClouds=^$|.*Public.*")
|
||||
```
|
||||
|
||||
#### Replace/Modify/Append Values
|
||||
|
||||
Replacements for values can be passed to the matrix as an array of strings, each matching the format of `<keyRegex>=<valueRegex>/<replacementValue>`.
|
||||
The replace argument will find any permutations where the key fully matches the key regex and the value fully matches the value regex, and replace the value with
|
||||
the replacement specified.
|
||||
|
||||
NOTE:
|
||||
- The replacement value supports regex capture groups, enabling substring transformations, e.g. `Foo=(.*)-replaceMe/$1-replaced`. See the below examples for usage.
|
||||
- For each key/value, the first replacement provided that matches will be the only one applied.
|
||||
- If `=` or `/` characters need to be part of the regex or replacement, escape them with `\`.
|
||||
|
||||
For example, given a matrix config like below:
|
||||
|
||||
```
|
||||
{
|
||||
"matrix": {
|
||||
"Agent": {
|
||||
"ubuntu-1804": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" }
|
||||
},
|
||||
"JavaTestVersion": [ "1.8", "1.11" ]
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
The normal matrix output (without replacements), looks like:
|
||||
|
||||
```
|
||||
$ ./Create-JobMatrix.ps1 -ConfigPath <test> -Selection all
|
||||
{
|
||||
"ubuntu1804_18": {
|
||||
"OSVmImage": "MMSUbuntu18.04",
|
||||
"Pool": "azsdk-pool-mms-ubuntu-1804-general",
|
||||
"JavaTestVersion": "1.8"
|
||||
},
|
||||
"ubuntu1804_111": {
|
||||
"OSVmImage": "MMSUbuntu18.04",
|
||||
"Pool": "azsdk-pool-mms-ubuntu-1804-general",
|
||||
"JavaTestVersion": "1.11"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Passing in multiple replacements, the output will look like below. Note that replacing key/values that appear nested within a grouping
|
||||
will not affect that segment of the job name, since the job takes the grouping name (in this case "ubuntu1804").
|
||||
|
||||
The below example includes samples of regex grouping references, and wildcard key/value regexes:
|
||||
|
||||
```
|
||||
$ $replacements = @('.*Version=1.11/2.0', 'Pool=(.*ubuntu.*)-general/$1-custom')
|
||||
$ ../Create-JobMatrix.ps1 -ConfigPath ./test.Json -Selection all -Replace $replacements
|
||||
{
|
||||
"ubuntu1804_18": {
|
||||
"OSVmImage": "MMSUbuntu18.04",
|
||||
"Pool": "azsdk-pool-mms-ubuntu-1804-custom",
|
||||
"JavaTestVersion": "1.8"
|
||||
},
|
||||
"ubuntu1804_20": {
|
||||
"OSVmImage": "MMSUbuntu18.04",
|
||||
"Pool": "azsdk-pool-mms-ubuntu-1804-custom",
|
||||
"JavaTestVersion": "2.0"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### NonSparseParameters
|
||||
|
||||
Sometimes it may be necessary to generate a sparse matrix, but keep the full combination of a few parameters. The
|
||||
NonSparseParameters argument allows for more fine-grained control of matrix generation. For example:
|
||||
|
||||
```
|
||||
./Create-JobMatrix.ps1 `
|
||||
-ConfigPath /path/to/matrix.json `
|
||||
-Selection sparse `
|
||||
-NonSparseParameters @("JavaTestVersion")
|
||||
```
|
||||
|
||||
Given a matrix like below with `JavaTestVersion` marked as a non-sparse parameter:
|
||||
|
||||
```
|
||||
{
|
||||
"matrix": {
|
||||
"Agent": {
|
||||
"windows-2022": { "OSVmImage": "MMS2022", "Pool": "azsdk-pool-mms-win-2022-general" },
|
||||
"ubuntu-1804": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" },
|
||||
"macos-11": { "OSVmImage": "macos-11", "Pool": "Azure Pipelines" }
|
||||
},
|
||||
"JavaTestVersion": [ "1.8", "1.11" ],
|
||||
"AZURE_TEST_HTTP_CLIENTS": "netty",
|
||||
"ArmTemplateParameters": [ "@{endpointType='storage'}", "@{endpointType='cosmos'}" ]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
A matrix with 6 entries will be generated: A sparse matrix of Agent, AZURE_TEST_HTTP_CLIENTS and ArmTemplateParameters
|
||||
(3 total entries) will be multipled by the two `JavaTestVersion` parameters `1.8` and `1.11`.
|
||||
|
||||
NOTE: NonSparseParameters are also applied when generating an imported matrix.
|
||||
|
||||
#### Under the hood
|
||||
|
||||
The script generates an N-dimensional matrix with dimensions equal to the parameter array lengths. For example,
|
||||
the below config would generate a 2x2x1x1x1 matrix (five-dimensional):
|
||||
|
||||
```
|
||||
"matrix": {
|
||||
"framework": [ "net461", "netcoreapp2.1" ],
|
||||
"additionalTestArguments": [ "", "/p:SuperTest=true" ]
|
||||
"pool": [ "ubuntu-18.04" ],
|
||||
"container": [ "ubuntu-18.04" ],
|
||||
"testMode": [ "Record" ]
|
||||
}
|
||||
```
|
||||
|
||||
The matrix is stored as a one-dimensional array, with a row-major indexing scheme (e.g. `(2, 1, 0, 1, 0)`).
|
||||
|
||||
## Testing
|
||||
|
||||
The matrix functions can be tested using [pester](https://pester.dev/). The test command must be run from within the tests directory.
|
||||
|
||||
```
|
||||
$ cd tests
|
||||
$ Invoke-Pester
|
||||
|
||||
Starting discovery in 3 files.
|
||||
Discovery finished in 75ms.
|
||||
[+] /home/ben/sdk/azure-sdk-tools/eng/common/scripts/job-matrix/tests/job-matrix-functions.filter.tests.ps1 750ms (309ms|428ms)
|
||||
[+] /home/ben/sdk/azure-sdk-tools/eng/common/scripts/job-matrix/tests/job-matrix-functions.modification.tests.ps1 867ms (250ms|608ms)
|
||||
[+] /home/ben/sdk/azure-sdk-tools/eng/common/scripts/job-matrix/tests/job-matrix-functions.tests.ps1 2.71s (725ms|1.93s)
|
||||
Tests completed in 4.33s
|
||||
Tests Passed: 141, Failed: 0, Skipped: 4 NotRun: 0
|
||||
```
|
||||
`azure-sdk-tools/doc/common/matrix_generator.md`
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче