azure-rest-api-specs-1/documentation/sdkautomation

SDK Automation Customization

This is the specification of the new SDK Automation customization configuration. Old customization that hardcoded in sdk automation will still work but this new approach is preferred.

SDK Automation workflow

Opened PR Validation Trigger

SDK Automation is launched with matrix in azure pipeline. For each language configured:

1. Get the PR merge commit and clone the spec repo on the merge commit.

2. Get the PR changed file list. For each changed file, find the nearest readme.md in parent folder. Get list of related readme.md.

3. Filter which sdk will be generated:

   1. For Swagger PR, filter the list of readme.md with: find the swagger-to-sdk section in the readme.md, and see if the specified language is configured for that readme.md. Example of swagger-to-sdk in SDK Automation:

      ```yaml $(swagger-to-sdk)
      swagger-to-sdk:
        - repo: azure-sdk-for-python
        - repo: azure-sdk-for-java
        - repo: azure-sdk-for-go
        - repo: azure-sdk-for-js
      ``` <EOL>
      

   2. For TypeSpec PR, filter the list of tspconfig.yaml: find the options config in tspconfig.yaml, and see if the specified language is configure for that tsp-location.yaml.

   If the configured language is not found here, generation for this typespec project will be skipped.

4. Get specificationRepositoryConfiguration.json from spec repo default branch. See SpecRepoConfig. Get the repo and branch config in the file.

5. Clone mainRepository and checkout mainBranch. If secondaryRepository is specified then checkout secondaryRepository and secondaryBranch instead.

6. Get swagger_to_sdk_config.json from cloned SDK repository. The config file path could be customized by configFilePath in spec config. For the definition of the config see SwaggerToSdkConfig.

7. Launch initScript defined in SwaggerToSdkConfig. All the script's working directory is root folder of cloned SDK repository.

8. Calculate PR diff and related readme.md. If generationCallMode is one-for-all-configs then run one pass for the rest steps, else (one-per-config) loop the rest steps with each readme.md.

9. Launch generateScript defined in SwaggerToSdkConfig with generateInput.json. The script should produce generateOutput.json if parseGenerateOutput is true. If dryRun is set to true then first run of generateScript will be used to collect package information , then loop each package info and checkout package related branch and launch generateScript with package related readmeMd and dryRun set to false.

10. Get generated package. If packageFolderFromFileSearch is defined with file search then package folder is detected based on git diff in SDK repository and algorithm described in SwaggerToSdkConfig Schema. Else package folder is from generateOutput.json. For each package loop the rest steps.

11. Launch buildScript to build the package. Collect the artifacts by config artifactPathFromFileSearch. This step could be skipped if it's not defined in SwaggerToSdkConfig and it's covered by generateScript and the result could be found in generateOutput.json.

12. Upload all the package related artifacts to Azure Storage Blob Container. All the artifact for one package is uploaded in one folder. These file could be downloaded on URL prefixed by downloadUrlPrefix defined in InstallInstructionScriptInput. It's redirected by openapiHub by design, and for SDK Automation on public repo the redirect don't need auth, but for SDK Automation in private repo it requires microsoft AAD auth. User could authenticate and download via web page oauth in browser or bearer token auth with az rest --resource in command line.

13. Launch changelogScript to get changelog and detect breaking change. This step could be skipped if changelog and breaking change could be found in generateOutput.json. If breaking change is found, the spec PR will be labelled with CI-BreakingChange-<Lang>.

14. Launch installInstructionScript to get install instruction for that package. This step could be skipped if install instruction could be found in generateOutput.json. The lite install instruction will be shown in spec PR comment, the full install instruction will be shown in generated SDK PR.

15. Commit the package related code in SDK repository. Force push to GenerationBranch in integrationRepository. Create or update GenerationPR from GenerationBranch to MainBranch in integrationRepository. If integrationRepository is a fork of mainRepository, its MainBranch should be synced once a day.

Continuous Integration (PR Merged) Trigger

Almost the same as opened PR trigger, with different on step 15:

15. Commit the package related code in SDK repository. Close GenerationPR and delete GenerationBranch. Force push to IntegrationBranch in integrationRepository. Create or update IntegrationPR from IntegrationBranch to MainBranch in mainRepository. Close the integrationPR if closeIntegrationPR in SwaggerToSdkConfig is set to true.

Definitions

SpecRepoConfig

This is type of file ./specificationRepositoryConfiguration.json in swagger spec repo.

SpecRepoConfig Example

{
  "sdkRepositoryMappings": {
    "azure-sdk-for-js": {
      "integrationRepository": "AzureSDKAutomation/azure-sdk-for-js",
      "mainRepository": "Azure/azure-sdk-for-js"
    },
    "azure-sdk-for-python": {
      "integrationRepository": "AzureSDKAutomation/azure-sdk-for-python",
      "mainRepository": "Azure/azure-sdk-for-python",
      "mainBranch": "release/v3"
    },
    "azure-sdk-for-python-track2": {
      "integrationRepository": "AzureSDKAutomation/azure-sdk-for-python",
      "mainRepository": "Azure/azure-sdk-for-python"
    },
    "azure-sdk-for-trenton": {
      "integrationRepository": "Azure/azure-sdk-for-trenton",
      "mainRepository": "Azure/azure-sdk-for-trenton",
      "secondaryRepository": "Azure/azure-sdk-for-trenton",
      "secondaryBranch": "secondary"
    }
  },
  "overrides": {
    "Azure/azure-rest-api-specs-pr": {
      "sdkRepositoryMappings": {
        "azure-sdk-for-js": {
          "integrationRepository": "azure-sdk/azure-sdk-for-js-pr",
          "mainRepository": "Azure/azure-sdk-for-js-pr"
        },
        "azure-sdk-for-python": {
          "integrationRepository": "azure-sdk/azure-sdk-for-python-pr",
          "mainRepository": "Azure/azure-sdk-for-python-pr",
          "mainBranch": "release/v3"
        },
        "azure-sdk-for-python-track2": {
          "integrationRepository": "azure-sdk/azure-sdk-for-python-pr",
          "mainRepository": "Azure/azure-sdk-for-python-pr"
        }
      }
    }
  }
}

SpecRepoConfig Schema

See ./SpecConfigSchema.json

SwaggerToSdkConfig

This is type of file ./swagger_to_sdk_config.json in sdk repo. The running environment of these scripts would be expected to be Ubuntu 18.04 on Azure Pipeline. This may change in the future. All the running script should be executable. The working folder of all the scripts is the root folder of sdk repo.

SwaggerToSdkConfig Example

{
  "advancedOptions": {
    "createSdkPullRequests": true,
    "generationCallMode": "one-for-all-configs"
  },
  "initOptions": {
    "initScript": {
      // Script to init dependencies.
      // Param: <path_to_initInput.json> <path_to_initOutput.json>
      // initInput.json: Not implemented. Placeholder for input arguments.
      // initOutput.json: See #initOutput.
      "path": "./eng/tools/sdk_init"
    }
  },
  "generateOptions": {
    // Param: <path_to_generateInput.json> <path_to_generateOutput.json>
    // generateInput.json: See #GenerateInput .
    // generateOutput.json: See #GenerateOutput .
    "generateScript": {
      "path": "./eng/tools/sdk_generate",
      "stderr": {
        "showInComment": true
      },
      "stdout": {
        // Show logs start with "[Autorest]" in PR comment.
        "showInComment": "^\\[Autorest\\]"
      }
    },

    "parseGenerateOutput": true
  },
  "packageOptions": {
    // Param: <path_to_package_folder>
    "buildScript": {
      "path": "./eng/tools/sdk_package",
      "stderr": {
        // Everything in stderr will show in comment and mark package with warning.
        "showInComment": true,
        "scriptWarning": true
      },
      "exitCode": {
        // If exit code is not zero, mark package with warning instead of error.
        "result": "warning"
      }
    },
    
    // Param: <path_to_package_folder>
    "changelogScript": {
      "path": "./eng/tools/sdk_breaking_change",
      "breakingChangeDetect": "Breaking Change"
    },

    "breakingChangeLabel": "CI-BreakingChange-DotNet"
  },
  "artifactOptions": {
    // Param: <path_to_installInstructionInput.json> <path_to_installInstructionOutput.json>
    // installInstructionInput.json: See #InstallInstructionScriptInput .
    // installInstructionOutput.json: See #InstallInstructionScriptOutput .
    "installInstructionScript": {
      "path": "./eng/tools/sdk_install_instruction"
    }
  }
}

SwaggerToSdkConfig Schema

See ./SwaggerToSdkConfigSchema.json

GenerateInput

Input file for generate script.

GenerateInput Example

{
  "dryRun": false,
  "specFolder": "/z/work/azure-rest-api-specs",
  "headSha": "fce3400431eff281bddd04bed9727e63765b8da0",
  "headRef": "refs/pull/1234/merge",
  "repoHttpsUrl": "https://github.com/Azure/azure-rest-api-specs.git",
  "trigger": "pull_request",
  "changedFiles": [
    "specification/cdn/something/cdn.json"
  ],
  "relatedReadmeMdFiles": [
    "specification/cdn/something/readme.md"
  ],
  "installInstructionInput": {
    "isPublic": false,
    "downloadUrlPrefix": "https://openapihub.test.azure-devex-tools.com/api/sdk-dl-pub?p=Azure/1234/azure-sdk-for-net/",
    "downloadCommandTemplate": "curl -L \"{URL}\" -o {FILENAME}",
    "trigger": "pullRequest"
  }
}

GenerateInput Schema

See ./GenerateInputSchema.json

GenerateOutput

Output file for generate script.

GenerateOutput Example

{
  "packages": [
    {
      "packageName": "Microsoft.Cdn",
      "path": [
        "sdk/cdn"
      ],
      "readmeMd": [
        "specification/cdn/something/readme.md"
      ],
      "changelog": {
        "content": "Feature: something \n Breaking Changes: something\n",
        "hasBreakingChange": true
      },
      "artifacts": [
        "sdk/cdn/cdn.nuget",
        "sdk/cdn/cdn.snuget"
      ],
      "installInstructions": {
        "full": "To install something...",
        "lite": "dotnet something"
      },
      "result": "succeeded"
    }
  ]
}

GenerateOutput Schema

See ./GenerateOutputSchema.json

InstallInstructionScriptInput

Input of install instruction script.

InstallInstructionScriptInput Example

{
  "packageName": "Microsoft.Cdn",
  "artifacts": [
    "sdk/cdn/cdn.nuget",
    "sdk/cdn/cdn.snuget"
  ],
  "isPublic": true,
  "downloadUrlPrefix": "https://portal.azure-devex-tools.com/api/sdk-dl-pub?p=Azure/azure-rest-api-specs/1234/azure-sdk-for-net/",
  "downloadCommandTemplate": "curl -L \"{URL}\" -o {FILENAME}",
  "trigger": "pullRequest",
}

InstallInstructionScriptInput Schema

See ./InstallInstructionScriptInput.json

InstallInstructionScriptOutput

Output of install instruction script.

InstallInstructionScriptOutput Example

{
  "full": "To install something...",
}

InstallInstructionScriptOutput Schema

See ./InstallInstructionScriptOutput.json

TriggerType

TriggerType Schema

{
  // How this generation is triggered.
  "$id": "TriggerType",
  "type": "string",
  "enum": ["pullRequest", "continuousIntegration"]
}

InitOutput

InitOutput Schema

{
  "type": "object",
  "properties": {
    "envs": {
      // Environment variable to be set in following scripts.
      "additionalProperties": {
        "type": "string"
      }
    }
  }
}