Azure
/
azureml-examples
зеркало из https://github.com/Azure/azureml-examples.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

Update resources name (#2521) 

* Update keyvault name

* Update attached compute name

* Fix if condition

* Update compute name
This commit is contained in:
Fred Li 2023-08-09 11:14:35 -07:00 коммит произвёл GitHub
Родитель 7d3a90c198
Коммит 2c8e615d87
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
18 изменённых файлов: 35 добавлений и 26 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

2
.github/workflows/cli-jobs-spark-attached-spark-pipeline-default-identity.yml поставляемый
Просмотреть файл

@ -52,7 +52,7 @@ jobs:
working-directory: cli
continue-on-error: true
run: |
bash -x jobs/spark/setup-attached-resources.sh resources/compute/attached-spark.yml
bash -x jobs/spark/setup-attached-resources.sh resources/compute/attached-spark.yml jobs/spark/attached-spark-pipeline-default-identity.yml
- name: run job
run: |
source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh";

2
.github/workflows/cli-jobs-spark-attached-spark-pipeline-managed-identity.yml поставляемый
Просмотреть файл

@ -57,7 +57,7 @@ jobs:
working-directory: cli
continue-on-error: true
run: |
bash -x jobs/spark/setup-attached-resources.sh resources/compute/attached-spark-system-identity.yml
bash -x jobs/spark/setup-attached-resources.sh resources/compute/attached-spark-system-identity.yml jobs/spark/attached-spark-pipeline-managed-identity.yml
- name: run job
run: |
source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh";

2
.github/workflows/cli-jobs-spark-attached-spark-pipeline-user-identity.yml поставляемый
Просмотреть файл

@ -52,7 +52,7 @@ jobs:
working-directory: cli
continue-on-error: true
run: |
bash -x jobs/spark/setup-attached-resources.sh resources/compute/attached-spark-user-identity.yml
bash -x jobs/spark/setup-attached-resources.sh resources/compute/attached-spark-user-identity.yml jobs/spark/attached-spark-pipeline-user-identity.yml
- name: run job
run: |
source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh";

2
.github/workflows/cli-jobs-spark-attached-spark-standalone-default-identity.yml поставляемый
Просмотреть файл

@ -52,7 +52,7 @@ jobs:
working-directory: cli
continue-on-error: true
run: |
bash -x jobs/spark/setup-attached-resources.sh resources/compute/attached-spark.yml
bash -x jobs/spark/setup-attached-resources.sh resources/compute/attached-spark.yml jobs/spark/attached-spark-standalone-default-identity.yml
- name: run job
run: |
source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh";

2
.github/workflows/cli-jobs-spark-attached-spark-standalone-managed-identity.yml поставляемый
Просмотреть файл

@ -57,7 +57,7 @@ jobs:
working-directory: cli
continue-on-error: true
run: |
bash -x jobs/spark/setup-attached-resources.sh resources/compute/attached-spark-system-identity.yml
bash -x jobs/spark/setup-attached-resources.sh resources/compute/attached-spark-system-identity.yml jobs/spark/attached-spark-standalone-managed-identity.yml
- name: run job
run: |
source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh";

2
.github/workflows/cli-jobs-spark-attached-spark-standalone-user-identity.yml поставляемый
Просмотреть файл

@ -52,7 +52,7 @@ jobs:
working-directory: cli
continue-on-error: true
run: |
bash -x jobs/spark/setup-attached-resources.sh resources/compute/attached-spark-user-identity.yml
bash -x jobs/spark/setup-attached-resources.sh resources/compute/attached-spark-user-identity.yml jobs/spark/attached-spark-standalone-user-identity.yml
- name: run job
run: |
source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh";

2
cli/jobs/spark/attached-spark-pipeline-default-identity.yml
Просмотреть файл

@ -20,4 +20,4 @@ jobs:
path: azureml://datastores/workspaceblobstore/paths/data/wrangled/
mode: direct
compute: mysparkcompute
compute: <ATTACHED_SPARK_POOL_NAME>

2
cli/jobs/spark/attached-spark-pipeline-managed-identity.yml
Просмотреть файл

@ -23,4 +23,4 @@ jobs:
identity:
type: managed
compute: mysparkcompute
compute: <ATTACHED_SPARK_POOL_NAME>

2
cli/jobs/spark/attached-spark-pipeline-user-identity.yml
Просмотреть файл

@ -23,4 +23,4 @@ jobs:
identity:
type: user_identity
compute: mysparkcompute
compute: <ATTACHED_SPARK_POOL_NAME>

2
cli/jobs/spark/attached-spark-standalone-default-identity.yml
Просмотреть файл

@ -29,4 +29,4 @@ args: >-
--titanic_data ${{inputs.titanic_data}}
--wrangled_data ${{outputs.wrangled_data}}
compute: mysparkcompute
compute: <ATTACHED_SPARK_POOL_NAME>

2
cli/jobs/spark/attached-spark-standalone-managed-identity.yml
Просмотреть файл

@ -32,4 +32,4 @@ args: >-
identity:
type: managed
compute: mysparkcompute
compute: <ATTACHED_SPARK_POOL_NAME>

2
cli/jobs/spark/attached-spark-standalone-user-identity.yml
Просмотреть файл

@ -32,4 +32,4 @@ args: >-
identity:
type: user_identity
compute: mysparkcompute
compute: <ATTACHED_SPARK_POOL_NAME>

17
cli/jobs/spark/setup-attached-resources.sh
Просмотреть файл

@ -30,15 +30,24 @@ az role assignment create --role "Storage Blob Data Owner" --assignee $AML_USER_
az synapse spark pool create --name $SPARK_POOL_NAME --workspace-name $SYNAPSE_WORKSPACE_NAME --resource-group $RESOURCE_GROUP --spark-version 3.2 --node-count 3 --node-size Medium --min-node-count 3 --max-node-count 10 --enable-auto-scale true
az synapse workspace firewall-rule create --name allowAll --workspace-name $SYNAPSE_WORKSPACE_NAME --resource-group $RESOURCE_GROUP --start-ip-address 0.0.0.0 --end-ip-address 255.255.255.255
TEMP_COMPUTE_FILE="temp-compute-setup.yml"
cp $1 $TEMP_COMPUTE_FILE
if [[ "$2" == *"managed-identity"* ]]
then
ATTACHED_COMPUTE_NAME=${ATTACHED_COMPUTE_NAME}mi
elif [[ "$2" == *"user-identity"* ]]
then
ATTACHED_COMPUTE_NAME=${ATTACHED_COMPUTE_NAME}ui
fi
sed -i "s/<SUBSCRIPTION_ID>/$SUBSCRIPTION_ID/g;
s/<RESOURCE_GROUP>/$RESOURCE_GROUP/g;
s/<SYNAPSE_WORKSPACE_NAME>/$SYNAPSE_WORKSPACE_NAME/g;
s/<SPARK_POOL_NAME>/$SPARK_POOL_NAME/g;
s/<AML_USER_MANAGED_ID>/$AML_USER_MANAGED_ID/g;" $TEMP_COMPUTE_FILE
s/<AML_USER_MANAGED_ID>/$AML_USER_MANAGED_ID/g;
s/<ATTACHED_SPARK_POOL_NAME>/$ATTACHED_COMPUTE_NAME/g;" $1
az ml compute attach --file $TEMP_COMPUTE_FILE --subscription $SUBSCRIPTION_ID --resource-group $RESOURCE_GROUP --workspace-name $AML_WORKSPACE_NAME
sed -i "s/<ATTACHED_SPARK_POOL_NAME>/$ATTACHED_COMPUTE_NAME/g;" $2
az ml compute attach --file $1 --subscription $SUBSCRIPTION_ID --resource-group $RESOURCE_GROUP --workspace-name $AML_WORKSPACE_NAME
az synapse role assignment create --workspace-name $SYNAPSE_WORKSPACE_NAME --role $SPARK_POOL_ADMIN_ROLE_ID --assignee $AML_USER_MANAGED_ID_OID
COMPUTE_MANAGED_IDENTITY=$(az ml compute show --name $ATTACHED_COMPUTE_NAME --resource-group $RESOURCE_GROUP --workspace-name $AML_WORKSPACE_NAME --query identity.principal_id --out tsv)

12
cli/readme.py
Просмотреть файл

@ -471,7 +471,7 @@ jobs:
working-directory: cli
continue-on-error: true\n"""
if is_spark_sample:
workflow_yaml += get_spark_setup_workflow(job)
workflow_yaml += get_spark_setup_workflow(job, posix_project_dir, filename)
workflow_yaml += f""" - name: run job
run: |
source "{GITHUB_WORKSPACE}/infra/bootstrapping/sdk_helpers.sh";
@ -862,7 +862,7 @@ def get_endpoint_name(filename, hyphenated):
return endpoint_name
def get_spark_setup_workflow(job):
def get_spark_setup_workflow(job, posix_project_dir, filename):
is_attached = "attached-spark" in job
is_user_identity = "user-identity" in job
is_managed_identity = "managed-identity" in job
@ -876,7 +876,7 @@ def get_spark_setup_workflow(job):
workflow += f""" - name: setup identities
run: |
bash -x setup-identities.sh
working-directory: cli/jobs/spark
working-directory: cli/{posix_project_dir}
continue-on-error: true\n"""
if is_attached:
workflow += f""" - name: setup attached spark
@ -885,15 +885,15 @@ def get_spark_setup_workflow(job):
if is_attached and is_user_identity:
workflow += f"""
run: |
bash -x jobs/spark/setup-attached-resources.sh resources/compute/attached-spark-user-identity.yml\n"""
bash -x {posix_project_dir}/setup-attached-resources.sh resources/compute/attached-spark-user-identity.yml {posix_project_dir}/{filename}.yml\n"""
if is_attached and is_managed_identity:
workflow += f"""
run: |
bash -x jobs/spark/setup-attached-resources.sh resources/compute/attached-spark-system-identity.yml\n"""
bash -x {posix_project_dir}/setup-attached-resources.sh resources/compute/attached-spark-system-identity.yml {posix_project_dir}/{filename}.yml\n"""
if is_attached and is_default_identity:
workflow += f"""
run: |
bash -x jobs/spark/setup-attached-resources.sh resources/compute/attached-spark.yml\n"""
bash -x {posix_project_dir}/setup-attached-resources.sh resources/compute/attached-spark.yml {posix_project_dir}/{filename}.yml\n"""
return workflow

2
cli/resources/compute/attached-spark-system-identity.yml
Просмотреть файл

@ -1,5 +1,5 @@
# attached-spark-system-identity.yaml
name: mysparkcompute
name: <ATTACHED_SPARK_POOL_NAME>
type: synapsespark

2
cli/resources/compute/attached-spark-user-identity.yml
Просмотреть файл

@ -1,5 +1,5 @@
# attached-spark-user-identity.yml
name: mysparkcompute
name: <ATTACHED_SPARK_POOL_NAME>
type: synapsespark

2
cli/resources/compute/attached-spark.yml
Просмотреть файл

@ -1,5 +1,5 @@
# attached-spark.yaml
name: mysparkcompute
name: <ATTACHED_SPARK_POOL_NAME>
type: synapsespark

2
sdk/python/jobs/spark/setup_spark.sh
Просмотреть файл

@ -88,7 +88,7 @@ then
ACCOUNT_KEY=$(az storage account keys list --account-name $AZURE_STORAGE_ACCOUNT --query "[0].value" -o tsv)
ACCESS_KEY_SECRET_NAME="autotestaccountkey"
KEY_VAULT_NAME="autotestsparkkv"
KEY_VAULT_NAME=${RESOURCE_GROUP}-kv
az keyvault create -n $KEY_VAULT_NAME -g $RESOURCE_GROUP
NOTEBOOK_PY="./data-wrangling/interactive_data_wrangling.py"