Simplifying the documentation (#87)
This commit is contained in:
Sushil Kumar
GitHub
Родитель
3ef9163310
Коммит
5df7dc9e54
11
README.md
11
README.md
|
|
@ -7,16 +7,11 @@ Disclaimer: The solution and samples provided in this repository is for learning
|
|||
|
||||
# Getting Started
|
||||
|
||||
Start by following the steps in the [deploy](/deploy/README.md) folder to setup the Azure resources required to build your pipeline.
|
||||
Start by following the [README.md](./deploy/README.md) to setup the Azure resources required to execute the pipeline.
|
||||
|
||||
Import the pipeline under the `workflow` folder to your Azure Synapse Analytics instance's workspace. Alternatively, you can copy the files to your repository (git or Azure DevOps) and link the repository to your Azure Synapse Analytics workspace.
|
||||
This solution uses the [Custom Vision Model](/src/aimodels) as a sample AI model to demonstrate end-to-end Azure Synapse workflow for geospatial analysis. In this sample solution, the AI model detects swimming pools for a given geospatial data.
|
||||
|
||||
Sample pipelines are provided that include the following AI Model:
|
||||
|
||||
### a. AI model
|
||||
|
||||
This solution uses the [Custom Vision Model](/src/aimodels) as a sample AI model for demonstrating end to end Azure Synapse workflow geospatial analysis. This sample solution uses Custom Vision model to detect pools in a given geospatial data.
|
||||
You can use any other AI model for object detection or otherwise to run against this solution with a similar [specification](/src/aimodels/custom_vision_object_detection_offline/specs/custom_vision_object_detection.json) or different specification as defined by AI model to integrate in your solution.
|
||||
You can use this solution to integrate other AI models. Each AI Model requires their input geospatial data to be in a specific format. When adapting this solution for a different AI Model, make sure the geospatial data transform steps are modified to adapt to the individual AI Model's needs.
|
||||
|
||||
## Contributing
|
||||
|
||||
|
|
|
|||
|
|
@ -8,6 +8,10 @@ parameters:
|
|||
displayName: Keyvault Name
|
||||
type: string
|
||||
default: cipipelineresources
|
||||
- name: AZURE_REGION
|
||||
displayName: Azure Region
|
||||
type: string
|
||||
default: eastus
|
||||
|
||||
variables:
|
||||
- name: serviceConnectionToAzure
|
||||
|
|
@ -54,6 +58,7 @@ jobs:
|
|||
set -e
|
||||
|
||||
export KEYVAULT_NAME=${{ parameters.KEYVAULT_NAME }}
|
||||
export AZURE_REGION=${{ parameters.AZURE_REGION }}
|
||||
|
||||
export AAD_GROUP_PRINCIPAL_ID="$(az keyvault secret show --vault ${KEYVAULT_NAME} --name AAD-GROUP-PRINCIPAL-ID --query value -o tsv)"
|
||||
export CI_BATCH_ACCOUNT_NAME="$(az keyvault secret show --vault ${KEYVAULT_NAME} --name CI-BATCH-ACCOUNT-NAME --query value -o tsv)"
|
||||
|
|
@ -69,7 +74,9 @@ jobs:
|
|||
export POSTGRES_ADMIN_LOGIN_PASS=$({ </dev/urandom LC_ALL=C grep -ao '[A-Za-z0-9]' | head -n$((RANDOM % 8 + 9)); echo ${chars:$((RANDOM % ${#chars})):1}; } | shuf | tr -d '\n')
|
||||
|
||||
# Setup the infrastructure for testing the pipeline
|
||||
DEPLOY_PGSQL=${USE_POSTGRE_SQL} $(System.DefaultWorkingDirectory)/deploy/setup.sh ${ENV_CODE} eastus custom-vision-model ${ENV_CODE}-environment ${CI_BATCH_ACCOUNT_NAME}
|
||||
DEPLOY_PGSQL=${USE_POSTGRE_SQL} \
|
||||
PRE_PROVISIONED_AI_MODEL_INFRA_NAME=${CI_BATCH_ACCOUNT_NAME} \
|
||||
$(System.DefaultWorkingDirectory)/deploy/setup.sh ${ENV_CODE} ${AZURE_REGION} ${ENV_CODE}-ci-environment
|
||||
|
||||
# Grant access on Synapse workspace to resource owner azure account
|
||||
SYNAPSE_WORKSPACE_NAME=$(az synapse workspace list --query "[?tags.workspaceId && tags.workspaceId == 'default'].name" -o tsv -g "${ENV_CODE}-pipeline-rg")
|
||||
|
|
|
|||
426
deploy/README.md
426
deploy/README.md
|
|
@ -1,391 +1,89 @@
|
|||
# Overview of Infrastructure deployment and configuration
|
||||
|
||||
We are using bicep templates to deploy the infrastructure resources.
|
||||
In a default setup we will be using following azure services:
|
||||
- Azure Batch Account
|
||||
- Azure Synapse Analytics
|
||||
- Azure Keyvault
|
||||
- Azure Storage Account
|
||||
- Azure Networking
|
||||
- Azure Log Analytics
|
||||
|
||||
# Prerequisites
|
||||
|
||||
The deployment script uses following tools, please follow the links provided to install the suggested tools on your computer using which you would execute the script.
|
||||
|
||||
- [bicep](https://docs.microsoft.com/azure/azure-resource-manager/bicep/install)
|
||||
- [git](https://github.com/git-guides/install-git)
|
||||
- [az cli](https://docs.microsoft.com/cli/azure/install-azure-cli)
|
||||
- [docker cli](https://docs.docker.com/get-docker/)
|
||||
- [jq](https://stedolan.github.io/jq/download/)
|
||||
|
||||
- The scripts are executed on bash shell, so if using a computer with windows based operating system, install a [WSL](https://docs.microsoft.com/windows/wsl/about) environment to execute the script.
|
||||
- Install bicep on az cli using command `az bicep install` and ensure the bicep version is >=0.8.9.
|
||||
Alternatively, one can use Azure Cloud Bash to deploy this sample solution in their Azure subscription.
|
||||
|
||||
- The user performing the deployment of the bicep template and the associated scripts should have `Owner` role assigned at the subscription to which the resources are being deployed.
|
||||
- The bicep templates have been written to adhere to the syntax and rules for bicep version >= 0.8.9. Please check your bicep version using `az bicep version` or `bicep --version` if you run into bicep related errors.
|
||||
- This solution assumes no interference from Policies deployed to your tenant preventing resources from being deployed.
|
||||
## RBAC requirement
|
||||
|
||||
The user performing the deployment of the bicep template and the associated scripts should have `Owner` role assigned at the subscription to which the resources are being deployed. This is needed in order to grant IAM roles to managed identities in bicep templates.
|
||||
|
||||
> [!NOTE]
|
||||
If you have started the deployment with a different role Ex: `Contributor`, and the deployment failed due to insufficient access. Please change the role to `Owner` and refresh the credentials by re-logging before attempting to deploy again.
|
||||
|
||||
## Infrastructure Deployment
|
||||
|
||||
Get the repository to find the scripts. Clone the repository using following command.
|
||||
|
||||
- Get the repository to find the scripts. Clone the repository using following command.
|
||||
```bash
|
||||
git clone git@github.com:Azure/Azure-Orbital-Analytics-Samples.git
|
||||
```
|
||||
|
||||
One would need [git](https://github.com/git-guides/install-git) cli tool to download the repository.
|
||||
|
||||
Alternatively, you can use Azure Cloud Bash to deploy this sample solution to your Azure subscription.
|
||||
|
||||
# How does scripts work?
|
||||
|
||||
The shell script runs an `az cli` command to invoke `bicep` tool.
|
||||
|
||||
This command recieves the bicep template as input, and converts the bicep templates into an intermediate ARM template output which is then submitted to Azure APIs to create the Azure resources.
|
||||
|
||||
# Overview of deployment and configuration
|
||||
|
||||
The deployment involves the following steps outlined below:
|
||||
|
||||
No | Step | Duration (approx.) | Required / Optional
|
||||
---|------|----------|---------------------
|
||||
1 | Preparing to execute the script | 1 minute | required
|
||||
2 | Deployment of Infrastructure using bicep template | 10 minutes | required
|
||||
3 | Configuring the Resources | 5 minutes | required
|
||||
4 | Packaging the Synapse pipline (optional) | 2 minutes | optional
|
||||
5 | Importing from Git Repository (optional) | 5 minutes | optional
|
||||
6 | Verifying infrastructure resources | 5 minutes | required
|
||||
7 | Load the Custom Vision Model to your Container Registry (optional) | 10 minutes | optional
|
||||
|
||||
Steps 2 through 4 can instead be deployed using a single script below:
|
||||
|
||||
```bash
|
||||
./deploy/setup.sh <environmentCode> <location> <pipelineName> <envTag>
|
||||
|
||||
```
|
||||
or
|
||||
|
||||
```bash
|
||||
./deploy/setup.sh <environmentCode> <location> <pipelineName> <envTag> <preprovisionedBatchAccountName> <deployPgsql>
|
||||
|
||||
```
|
||||
|
||||
If you like to package other pipelines or re-package an updated/modified pipeline, follow the instructions under `Packaging the Synapse pipeline` section. The script mentioned in that section can be rerun multiple times.
|
||||
|
||||
Arguments | Required | Type | Sample value
|
||||
----------|-----------|-------|-------
|
||||
environmentCode | yes | string | aoi
|
||||
location | yes | string | westus
|
||||
pipelineName | no | string | Allowed value: custom-vision-model
|
||||
envTag | no | string | synapse\-\<environmentCode\>
|
||||
preprovisionedBatchAccountName | no | string | aoibatchaccount
|
||||
deployPgsql | no | boolean | false
|
||||
|
||||
**Note**: If you do not pass the optional pipelineName parameter value, no zip file will be generated. You may however run the `package.sh` script to generate a zip file after running the `setup.sh` script to generate the zip file.
|
||||
|
||||
## Preparing to execute the script
|
||||
|
||||
Before executing the script one would need to login to azure using `az` cli and set the correct subscription in which they want to provision the resources.
|
||||
Before executing the script, please login to azure using `az` cli and set the subscription in which you want to provision the resources.
|
||||
|
||||
```bash
|
||||
az login
|
||||
az account set -s <subscription_id>
|
||||
```
|
||||
|
||||
Script has been written to be executed with minimalistic input, it requires following input
|
||||
- `environmentCode` which serves as the prefix for infrastructure services names. Allows only alpha numeric(no special characters) and must be between 3 and 8 characters.
|
||||
- `location` which suggests which azure region infrastructure is deployed in.
|
||||
- `environmentTag` / `envTag` serves as a simple label / tag to all resources being deployed as part of the bicep template to your subscription.
|
||||
- `pipelineName` refers to the name of the pipeline that is to be package for deployment to your Synapse Workspace. Allowed value is custom-vision-model.
|
||||
|
||||
## Deployment of Infrastructure using bicep template
|
||||
|
||||
If you have deployed the solution using `setup.sh` script, you should skip this step. However, if you have not run the `setup.sh` script, the steps outlined in this section are required.
|
||||
|
||||
To install infrastructure execute install.sh script as follows
|
||||
The following command will install, configure and generate the custom vision model package.
|
||||
|
||||
```bash
|
||||
./deploy/install.sh <environmentCode> <location> <envTag>
|
||||
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```bash
|
||||
./deploy/install.sh <environmentCode> <location> <envTag> <deploymentName> <deployBatchAccount> <deployPgsql>
|
||||
|
||||
```
|
||||
|
||||
You will be prompted to enter in a password for postgres. If left empty, an auto generated password will be created and stored in the keyvault. The password requirements are listed below:
|
||||
- Between 8 characters and 128 characters
|
||||
- Must contain characters from three of the following categories – English uppercase letters, English lowercase letters, numbers (0-9), and non-alphanumeric characters (!, $, #, %, etc.).
|
||||
- Your password cannot contain all or part of the login name. Part of a login name is defined as three or more consecutive alphanumeric characters.
|
||||
|
||||
Default values for the parameters are provided in the script itself.
|
||||
|
||||
Arguments | Required | Type | Sample value
|
||||
----------|-----------|-------|------------
|
||||
environmentCode | yes | string | aoi
|
||||
location | yes | string |westus
|
||||
envTag | no | string | synapse\-\<environmentCode\>
|
||||
deploymentName | no | string | aoi-deploy
|
||||
deployBatchAccount | no | boolean | false
|
||||
deployPgsql | no | boolean | false
|
||||
|
||||
|
||||
For eg.
|
||||
|
||||
```bash
|
||||
./deploy/install.sh aoi westus demo
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```bash
|
||||
./deploy/install.sh aoi westus demo aoi-deploy false false
|
||||
```
|
||||
|
||||
Users can also use bicep template directly instead of using the script `install.sh`
|
||||
|
||||
To deploy the resources using the bicep template use the command as follows:
|
||||
|
||||
```bash
|
||||
az deployment sub create -l <region_name> -n <deployment_name> -f main.bicep -p location=<region_name> environmentCode=<environment_name_prefix> environment=<tag_value>
|
||||
./deploy/setup.sh <environmentCode> <location>
|
||||
```
|
||||
- `environmentCode` used as a prefix for Azure resource names. It allows only alpha numeric(no special characters) and must be between 3 and 8 characters.
|
||||
- `location`is a valid Azure region.
|
||||
|
||||
For eg.
|
||||
```bash
|
||||
az deployment sub create -l <region> -n aoi -f main.bicep -p location=<region> environmentCode=aoi environment=synapse-aoi
|
||||
./deploy/setup.sh aoi eastus
|
||||
```
|
||||
|
||||
[setup.sh](./setup.sh) executes tasks in 3 steps
|
||||
- installs the infrastructure using [install.sh](./install.sh) script.
|
||||
- configures the infrastructure for setting up the dependecies using [configure.sh](./configure.sh) script.
|
||||
- packages the pipeline code to a zip file using [package.sh](./package.sh) script.
|
||||
|
||||
## Configuring the Resources
|
||||
After the script has run successfully, please check the batch-account pool created is created successfully.
|
||||
|
||||
If you have deployed the solution using `setup.sh` script, you should skip this step. However, if you have not run the `setup.sh` script, the steps outlined in this section are required.
|
||||
- You can find the batch account named as `batchact<10-character-random-string>` under the resource group `<environment-code>-orc-rg`.
|
||||
- Go to the Batch Account and switch to the pools blade. Look for one or more pools created by the bicep template. Make sure the resizing of the pool is completed without any errors.
|
||||
- Resizing may take a few minutes. Pools that are resizing are indicated by `0 -> 1` numbers under dedicated nodes column. Pools that have completed resizing should show the number of dedicated nodes.
|
||||
- Wait for all pools to complete resizing before moving to the next steps.
|
||||
|
||||
Next step is to configure your resources and set them up with the required dependencies like Python files, Library requirements and so on, before importing the Synapse pipeline. Run the `configure.sh` script below to perform the configuration:
|
||||
Error while resizing the pools are indicated by red exclamation icon next to the pool. Most common issues causing failure are related to the VM Quota limitations.
|
||||
|
||||
```bash
|
||||
./deploy/configure.sh <environmentCode>
|
||||
```
|
||||
or
|
||||
## Executing Synapse Pipeline
|
||||
|
||||
```bash
|
||||
./deploy/configure.sh <environmentCode> <batchAccountName> <batchAccountRG> <batchAccountKey> <batchAccountPoolName> <synapseWorkspaceRG> <synapseWorkspace> <synapsePool> <synapseStorageAccount>
|
||||
```
|
||||
We have published the Pipeline on Synapse gallery, follow the following steps to find the Synapse pipeline and execute.
|
||||
|
||||
Arguments | Required | Type | Sample value
|
||||
----------|-----------|-------|-------------
|
||||
environmentCode | yes | string | aoi
|
||||
batchAccountName | no | string | aoibatchaccount
|
||||
batchAccountRG | no | string | aoibatchaccountrg
|
||||
batchAccountKey | no | string | 2kfdof824lre-03k...d8kj4899==
|
||||
batchAccountPoolName | no | string | aoibatchaccountpool
|
||||
synapseWorkspaceRG | no | string | aoisynapseworkspacerg
|
||||
synapseWorkspace | no | string | aoisynapseworksapce
|
||||
synapsePool | no | string | synapsepoolname
|
||||
synapseStorageAccount | no | string | synrawdatastorage
|
||||
- Find your Synapse workspace named as `synws<10-character-random-string>` in resource group `<environment-code>-pipeline-rg`.
|
||||
- Click on `Workspace web URL` to reach the synapse workspace portal homepage.
|
||||
- Click on `Knowledge center`.
|
||||
- Click on `Browse gallery`.
|
||||
- Click on `Pipelines` in the top menu.
|
||||
- In the search box type `space` to find the pipeline named `Spaceborne Data Analysis Master Pipeline`.
|
||||
- Click on pipeline name to select it, and press `Continue` button at the end of the page.
|
||||
|
||||
Follow the steps in [instructions.md](./gallery/instructions.md) to configure and execute the pipeline.
|
||||
|
||||
## Packaging the Synapse Pipeline
|
||||
|
||||
You may repeat the steps outlined in this section multiple times to package the pipeline irrespective of whether you have already run the `package.sh` script or `setup.sh` script.
|
||||
|
||||
To package the Synapse pipeline, run the `package.sh` script by following the syntax below:
|
||||
|
||||
```bash
|
||||
./deploy/package.sh <environmentCode> <pipelineName>
|
||||
```
|
||||
or
|
||||
|
||||
```bash
|
||||
./deploy/package.sh <environmentCode> <pipelineName> <batchAccountName> <batchAccountRG> <batchAccountStorageAccountName> <keyVaultName> <rawStorageAccountRG> <rawStorageAccountName> <synapseWorkspaceRG> <synapseWorkspace> <synapseStorageAccount> <synapsePool> <deployPgsql>
|
||||
```
|
||||
|
||||
Arguments | Required | Type | Sample value
|
||||
----------|-----------|-------|---------------
|
||||
environmentCode | yes | string | aoi
|
||||
pipelineName | no | string | Allowed Value: custom-vision-model
|
||||
batchAccountName | no | string | aoibatchaccount
|
||||
batchAccountRG | no | string | aoibatchaccountrg
|
||||
batchAccountStorageAccountName | no | string | aoibatchstorageaccount
|
||||
keyVaultName | no | string | aoiKeyVault
|
||||
rawStorageAccountRG | no | string | aoi-data-rg
|
||||
rawStorageAccountName | no | string | rawdata34keh240
|
||||
synapseWorkspaceRG | no | string | aoisynapseworkspacerg
|
||||
synapseWorkspace | no | string | aoisynapseworksapce
|
||||
synapseStorageAccount | no | string | synrawdatastorage
|
||||
synapsePool | no | string | synapsepoolname
|
||||
deployPgsql | no | boolean | false
|
||||
|
||||
## Importing from Git Repository
|
||||
|
||||
Unzip the contents of the ZIP file generate by running the `package.sh` or `setup.sh` with pipelineName and use the contents to load them to your repository.
|
||||
|
||||
A few things to consider prior to integration of Git / GitHub Repository with Synapse Studio:
|
||||
|
||||
* Do not bring the files from [workflow folder](../src/workflow) directly into your repository that you will use to integrate with Synapse Studio. You will need to run `package.sh` or `setup.sh` to replace placeholders in one or more files before checking-in the files to your repository to be used with the Synapse Studio.
|
||||
* You can either create a new repository or use a forked version of the [Azure Orbital Analytics Sample](https://github.com/Azure/Azure-Orbital-Analytics-Samples) repository. If you use a new repository, use the Unzipped contents of the ZIP file to load into your new repository. If you use forked version of the [Azure Orbital Analytics Sample](https://github.com/Azure/Azure-Orbital-Analytics-Samples) repository, overwrite the contents of [Custom vision model workflow folder](../src/workflow/custom-vision-model) with the Unzipped contents.
|
||||
|
||||
|
||||
To import pipeline into the Synape Studio is through Source Control repository like GitHub or Azure DevOps repository, refer to the document on [Source Control](https://docs.microsoft.com/azure/synapse-analytics/cicd/source-control) to learn about Git Integration for Azure Synapse Analytics and how to setup.
|
||||
|
||||
**Note**: Once the Synapse Studio is linked to your repository, make sure you publish all the components. Failure to publish the components imported by linked Synapse Studio to Github / ADO repository, will result in errors when running the pipeline.
|
||||
|
||||
## Verifying infrastructure resources
|
||||
|
||||
Once setup has been executed one can check for following resource-groups and resources to confirm the successful execution.
|
||||
|
||||
Following is the list of resource-groups and resources that should be created if we executed the command `./deploy/install.sh aoi <region>`
|
||||
|
||||
- `aoi-data-rg`
|
||||
|
||||
This resource group houses data resources.
|
||||
|
||||
- Storage account named `rawdata<10-character-random-string>` to store raw input data for pipelines.
|
||||
- Keyvault named `kvd<10-character-random-string>` to store credentials as secrets.
|
||||
- Postgres Single Server DB named `pg<10-character-random-string>`
|
||||
|
||||
- `aoi-monitor-rg`
|
||||
|
||||
This resource group houses monitoring resources.
|
||||
|
||||
- App Insights instance named `aoi-monitor-appinsights` for monitoring.
|
||||
- Log Analytics workspace named `aoi-monitor-workspace` to store monitoring data.
|
||||
|
||||
- `aoi-network-rg`
|
||||
|
||||
This resource group houses networking resources.
|
||||
|
||||
- Virtual network named `aoi-vnet` which has 3 subnets.
|
||||
|
||||
- `pipeline-subnet`
|
||||
- `data-subnet`
|
||||
- `orchestration-subnet`
|
||||
- It also has a list security groups to restrict access on the network.
|
||||
|
||||
- `aoi-orc-rg`
|
||||
|
||||
This resource group houses pipeline orchestration resources.
|
||||
|
||||
- Storage account named `batchacc<10-character-random-string>` for batch account.
|
||||
- Batch Account named `batchact<10-character-random-string>`.
|
||||
|
||||
Also, go to the Batch Account and switch to the pools blade. Look for one or more pools created by the bicep template. Make sure the resizing of the pool is completed without any errors.
|
||||
|
||||
- Error while resizing the pools are indicated by red exclamation icon next to the pool. Most common issues causing failure are related to the VM Quota limitations.
|
||||
- Resizing may take a few minutes. Pools that are resizing are indicated by `0 -> 1` numbers under dedicated nodes column. Pools that have completed resizing should show the number of dedicated nodes.
|
||||
|
||||
Wait for all pools to complete resizing before moving to the next steps.
|
||||
|
||||
Note: The Bicep template adds the Synapse workspace's Managed Identity to the Batch Account as `Contributor`. Alternatively, Custom Role Definitions can be used to assign the Synapse workspace's Managed Identity to the Batch Account with required Azure RBAC operations.
|
||||
|
||||
- Keyvault named `kvo<10-character-random-string>`.
|
||||
- User managed identity `aoi-orc-umi` for access and authentication.
|
||||
- Azure Container registry instance named `acr<10-character-random-string>` to store container images.
|
||||
|
||||
- `aoi-pipeline-rg`
|
||||
|
||||
This resource group houses Synapse pipeline resources.
|
||||
|
||||
- Keyvault instance named `kvp<10-character-random-string>` to hold secrets for pipeline.
|
||||
- Storage account named `synhns<10-character-random-string>` for Synapse workspace.
|
||||
- Synapse workspace named `synws<10-character-random-string>` to hold pipeline resources.
|
||||
- Synapse spark pool `pool<10-character-random-string>` to run analytics.
|
||||
|
||||
|
||||
## Load the Custom Vision Model to your Container Registry
|
||||
|
||||
There are three ways to load an AI Model with this pipeline. Below are the three options.
|
||||
|
||||
Use one of the three options listed below. For option b and option c, either use `registry` property to pass credentials (requires update to the pipeline) or have your Batch Account pool configured with the ACR credentials when setting up the Batch Account pool.
|
||||
|
||||
a. Use the publicly hosted Custom Vision Model as GitHub Packages.
|
||||
|
||||
No additional steps are required for this approach. Custom Vision Model is containerized image that can be pulled from `docker pull ghcr.io/azure/azure-orbital-analytics-samples/custom_vision_offline:latest`. The [Specification document](../src/aimodels/custom_vision_object_detection_offline/specs/custom_vision_object_detection.json) in this repository already points to the publicly hosted GitHub Registry.
|
||||
|
||||
b. Download the publicly hosted Custom Vision Model and host it on your Container Registry.
|
||||
|
||||
Run the shell cmds below to pull and push the image to your Container Registry.
|
||||
|
||||
|
||||
```bash
|
||||
|
||||
docker pull ghcr.io/azure/azure-orbital-analytics-samples/custom_vision_offline:latest
|
||||
|
||||
docker tag ghcr.io/azure/azure-orbital-analytics-samples/custom_vision_offline:latest <container-registry-name>.azurecr.io/custom_vision_offline:latest
|
||||
|
||||
az acr login --name <container-registry-name>
|
||||
|
||||
docker push <container-registry-name>.azurecr.io/custom_vision_offline:latest
|
||||
|
||||
```
|
||||
|
||||
Update the `algImageName` value in [Specification document](../src/aimodels/custom_vision_object_detection_offline/specs/custom_vision_object_detection.json) to point to the new image location.
|
||||
|
||||
c. BYOM (Bring-your-own-Model) and host it on your Container Registry.
|
||||
|
||||
If you have the image locally, run the shell cmds below to push the image to your Container Registry.
|
||||
|
||||
```bash
|
||||
|
||||
docker tag custom_vision_offline:latest <container-registry-name>.azurecr.io/custom_vision_offline:latest
|
||||
|
||||
az acr login --name <container-registry-name>
|
||||
|
||||
docker push <container-registry-name>.azurecr.io/custom_vision_offline:latest
|
||||
|
||||
```
|
||||
Update the `algImageName` value in [Specification document](../src/aimodels/custom_vision_object_detection_offline/specs/custom_vision_object_detection.json) to point to the new image location.
|
||||
|
||||
Note: When using a private Container Registry, update `containerSettings` property in your [Custom Vision Object Detection](/src/workflow/pipeline/Custom%20Vision%20Object%20Detection.json) pipeline and add the following sub-property in order to authenticate to Container Registry :
|
||||
```json
|
||||
"registry": {
|
||||
"registryServer": "",
|
||||
"username": "",
|
||||
"password": ""
|
||||
}
|
||||
```
|
||||
|
||||
The above change will need to be made to the `Custom Vision Model Transform` pipeline. Look for activity named `Custom Vision` of type Web activity and update the body property (under Settings tab) for that activity.
|
||||
|
||||
[Specification document](../src/aimodels/custom_vision_object_detection_offline/specs/custom_vision_object_detection.json) and [Configuration file](../src/aimodels/custom_vision_object_detection_offline/config/config.json) required to run the Custom Vision Model.
|
||||
|
||||
- Specification document - This solution has a framework defined to standardized way of running AI Models as containerized solutions. A Specification document works as a contract definition document to run an AI Model.
|
||||
|
||||
- Configuration file - Each AI Model may require one or more parameters to run the model. This parameters driven by the end users are passed to the AI Model in the form of a configuration file. The schema of these configuration file is specific to the AI Model and hence we provide a template for the end user to plug-in their values.
|
||||
|
||||
# Running the pipeline (Custom Vision Model)
|
||||
|
||||
Before starting the pipeline, prepare the storage account in <environmentCode>-data-rg resource group by creating a container for the pipeline run.
|
||||
|
||||
- Create a new container for every pipeline run. Make sure the container name does not exceed 8 characters.
|
||||
|
||||
- Under the newly created container, add two folders. One folder named `config` with the following configuration files:
|
||||
|
||||
- [Specification document](../src/aimodels/custom_vision_object_detection_offline/specs/custom_vision_object_detection.json) configuration file that is provided by the AI Model partner.
|
||||
- [Config file](../src/aimodels/custom_vision_object_detection_offline/config/config.json) specific to the AI Model that contains parameters to be passed to the AI Model.
|
||||
coordinates.
|
||||
|
||||
Another folder named `raw` with sample Geotiff to be processed by the pipeline. You can use this [Geotiff file](https://aoigeospatial.blob.core.windows.net/public/samples/sample_4326.tif) hosted as a sample or any Geotiff with CRS of EPSG 4326.
|
||||
|
||||
When using this sample file, update your `AOI` parameter when kicking off the workflow with bbox value of `-117.063550 32.749467 -116.999386 32.812946`.
|
||||
|
||||
To run the pipeline, open the Synapse Studio for the Synapse workspace that you have created and follow the below listed steps.
|
||||
|
||||
- Open the `E2E Custom Vision Model Flow` and click on debug button
|
||||
|
||||
- When presented with the parameters, fill out the values. Below table provide the details on that each parameter represents if batch-account (by default) is provisioned in infrastructure setup.
|
||||
|
||||
| parameter | description |
|
||||
|--|--|
|
||||
| Prefix | This is the Storage container name created in [Running the pipeline section](#running-the-pipeline) that hosts the Raw data|
|
||||
| StorageAccountName | Name of the Storage Account in <environmentCode>-data-rg resource group that hosts the Raw data |
|
||||
| AOI | Area of Interest over which the AI Model is run |
|
||||
| BatchAccountName | Name of the Batch Account in <environmentCode>-orc-rg resource group to run the AI Model |
|
||||
| BatchJobName | Job name within the Batch Account in <environmentCode>-orc-rg resource group that runs the AI Model |
|
||||
| BatchLocation | Location of the Batch Account in <environmentCode>-orc-rg resource group that runs the AI Model |
|
||||
|
||||
- In case AKS (Azure Kubernetes Service) is provisioned, parameters and their descriptions are:
|
||||
|
||||
| parameter | description |
|
||||
|--|--|
|
||||
| Prefix | This is the Storage container name created in [Running the pipeline section](#running-the-pipeline) that hosts the Raw data|
|
||||
| StorageAccountName | Name of the Storage Account in <environmentCode>-data-rg resource group that hosts the Raw data |
|
||||
| AOI | Area of Interest over which the AI Model is run |
|
||||
| AksManagementRestApiURL | AKS Management Rest API Endpoint URL where Azure Synapse makes request calls to send kubectl commands to. Refer to [doc](https://docs.microsoft.com/en-us/rest/api/aks/managed-clusters/run-command). |
|
||||
| PersistentVolumeClaim | Persistent Volume Claim Name used for the AI-Model execution Kubernetes pod. This is preconfigured during setup and configuration, and can be found from Azure portal (provisioned AKS-> 'Storage'-> 'Persistent volume claims'). |
|
||||
|
||||
- Once the parameters are entered, click ok to submit and kick off the pipeline.
|
||||
|
||||
- Wait for the pipeline to complete.
|
||||
|
||||
# Cleanup Script
|
||||
## Cleanup Script
|
||||
|
||||
We have a cleanup script to cleanup the resource groups and thus the resources provisioned using the `environmentCode`.
|
||||
As discussed above the `environmentCode` is used as prefix to generate resource group names, so the cleanup-script deletes the resource groups with generated names.
|
||||
|
|
@ -412,24 +110,6 @@ NO_DELETE_PIPELINE_RESOURCE_GROUP=true
|
|||
./deploy/cleanup.sh <environmentCode>
|
||||
```
|
||||
|
||||
# Using different infrastructure to host AI-Model execution
|
||||
## AI Model
|
||||
|
||||
We have introduced the work to cater to the need of customers to use different infrastructure types to host the execution of AI-Models:
|
||||
|
||||
Previously, we had the AI-Model executing on Azure Batch Account, now we are extending this to use AKS.
|
||||
|
||||
To help users we have introduced a new variable `AI_MODEL_INFRA_TYPE` in [setup.sh](./setup.sh) to suggest which infrastructure type they would want to use.
|
||||
|
||||
`AI_MODEL_INFRA_TYPE` currently supports 2 values, with default being `batch-account`:
|
||||
- `batch-account`
|
||||
- `aks`
|
||||
|
||||
Those users requiring to execute AI-Model against `batch-account` will not need any changes to their commands, however, those who want to execute against `aks`, can simply execute by setting the environment variable `AI_MODEL_INFRA_TYPE`.
|
||||
|
||||
```bash
|
||||
AI_MODEL_INFRA_TYPE=aks ./deploy/setup.sh <environmentCode> <location> <pipelineName> <envTag>
|
||||
```
|
||||
|
||||
# Attributions And Disclaimers
|
||||
|
||||
- [Geotiff file](https://aoigeospatial.blob.core.windows.net/public/samples/sample_4326.tif) provided as sample are attributed to NAIP Imagery available via [Planetary Computer](https://planetarycomputer.microsoft.com) They are covered under [USDA](https://ngda-imagery-geoplatform.hub.arcgis.com)
|
||||
Follow the [document](./using-ai-model-in-pipeline.md) to understand and use [Custom Vision Model](/src/aimodels) or bring different ai-model for processing with the pipeline.
|
||||
|
|
|
|||
|
|
@ -5,10 +5,10 @@
|
|||
|
||||
ENV_CODE=${1:-${ENV_CODE}}
|
||||
LOCATION=${2:-${LOCATION}}
|
||||
PIPELINE_NAME=${3:-${PIPELINE_NAME:-"custom-vision-model"}}
|
||||
ENV_TAG=${4:-${ENV_TAG}}
|
||||
PRE_PROVISIONED_AI_MODEL_INFRA_NAME=${5:-$PRE_PROVISIONED_AI_MODEL_INFRA_NAME}
|
||||
AI_MODEL_INFRA_TYPE=${6:-${AI_MODEL_INFRA_TYPE:-"batch-account"}} # Currently supported values are aks and batch-account
|
||||
ENV_TAG=${3:-${ENV_TAG:-"synapse-$ENV_CODE"}}
|
||||
PIPELINE_NAME=${4:-${PIPELINE_NAME:-"custom-vision-model"}}
|
||||
AI_MODEL_INFRA_TYPE=${5:-${AI_MODEL_INFRA_TYPE:-"batch-account"}} # Currently supported values are aks and batch-account
|
||||
PRE_PROVISIONED_AI_MODEL_INFRA_NAME=${6:-$PRE_PROVISIONED_AI_MODEL_INFRA_NAME}
|
||||
DEPLOY_PGSQL=${7:-${DEPLOY_PGSQL:-"true"}}
|
||||
|
||||
set -ex
|
||||
|
|
|
|||
|
|
@ -0,0 +1,69 @@
|
|||
## Using AI-Models
|
||||
|
||||
There are three ways to load an AI Model with this pipeline. One can use one of the three methods listed below.
|
||||
|
||||
### Use the publicly hosted Custom Vision Model as GitHub Packages.
|
||||
|
||||
No additional steps are required for this approach. Custom Vision Model is containerized image that can be pulled from `docker pull ghcr.io/azure/azure-orbital-analytics-samples/custom_vision_offline:latest`.
|
||||
|
||||
The [Specification document](../src/aimodels/custom_vision_object_detection_offline/specs/custom_vision_object_detection.json) in this repository already points to the publicly hosted GitHub Registry.
|
||||
|
||||
### Host public Custom Vision Model on your Container Registry
|
||||
|
||||
In order to use this method one would need to do one of the following methods to get the ai-model docker image
|
||||
- use `registry` property to pass credentials (requires update to the pipeline)
|
||||
- Configure Batch Account pool with the ACR credentials when setting up the Batch Account pool.
|
||||
|
||||
Run following command to pull and push the image to your Container Registry.
|
||||
|
||||
```bash
|
||||
|
||||
docker pull ghcr.io/azure/azure-orbital-analytics-samples/custom_vision_offline:latest
|
||||
|
||||
docker tag ghcr.io/azure/azure-orbital-analytics-samples/custom_vision_offline:latest <container-registry-name>.azurecr.io/custom_vision_offline:latest
|
||||
|
||||
az acr login --name <container-registry-name>
|
||||
|
||||
docker push <container-registry-name>.azurecr.io/custom_vision_offline:latest
|
||||
|
||||
```
|
||||
|
||||
Update the `algImageName` value in [Specification document](../src/aimodels/custom_vision_object_detection_offline/specs/custom_vision_object_detection.json) to point to the new image location.
|
||||
|
||||
### BYOM (Bring-your-own-Model) and host it on your Container Registry.
|
||||
|
||||
In order to use this method one would need to do one of the following methods to get the ai-model docker image
|
||||
- use `registry` property to pass credentials (requires update to the pipeline)
|
||||
- Configure Batch Account pool with the ACR credentials when setting up the Batch Account pool.
|
||||
|
||||
If you have the image locally, run the shell cmds below to push the image to your Container Registry.
|
||||
|
||||
```bash
|
||||
|
||||
docker tag custom_vision_offline:latest <container-registry-name>.azurecr.io/custom_vision_offline:latest
|
||||
|
||||
az acr login --name <container-registry-name>
|
||||
|
||||
docker push <container-registry-name>.azurecr.io/custom_vision_offline:latest
|
||||
|
||||
```
|
||||
Update the `algImageName` value in [Specification document](../src/aimodels/custom_vision_object_detection_offline/specs/custom_vision_object_detection.json) to point to the new image location.
|
||||
|
||||
## Using Private Container Registry
|
||||
|
||||
When using a private Container Registry, update `containerSettings` property in your [Custom Vision Object Detection](/src/workflow/pipeline/Custom%20Vision%20Object%20Detection.json) pipeline and add the following sub-property in order to authenticate to Container Registry :
|
||||
```json
|
||||
"registry": {
|
||||
"registryServer": "",
|
||||
"username": "",
|
||||
"password": ""
|
||||
}
|
||||
```
|
||||
|
||||
The above change will need to be made to the `Custom Vision Model Transform` pipeline. Look for activity named `Custom Vision` of type Web activity and update the body property (under Settings tab) for that activity.
|
||||
|
||||
[Specification document](../src/aimodels/custom_vision_object_detection_offline/specs/custom_vision_object_detection.json) and [Configuration file](../src/aimodels/custom_vision_object_detection_offline/config/config.json) required to run the Custom Vision Model.
|
||||
|
||||
- Specification document - This solution has a framework defined to standardized way of running AI Models as containerized solutions. A Specification document works as a contract definition document to run an AI Model.
|
||||
|
||||
- Configuration file - Each AI Model may require one or more parameters to run the model. This parameters driven by the end users are passed to the AI Model in the form of a configuration file. The schema of these configuration file is specific to the AI Model and hence we provide a template for the end user to plug-in their values.
|
||||
|
|
@ -44,3 +44,6 @@ Solution:
|
|||
|
||||
After importing your pipeline from your repository or pipeline template approach, please use "Publish" or "Publish All" to publish your pipeline and its associated components like Linked Services.
|
||||
|
||||
## Policies interference
|
||||
|
||||
This solution assumes no interference from Policies deployed to your tenant preventing resources from being deployed.
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче