azureml-examples/cli/CONTRIBUTING.md

55 строки
2.6 KiB
Markdown
Исходник Постоянная ссылка Обычный вид История

# Contributing (CLI)
refactor for CLI 2.0 preview (#495) * scaffolding * fixes, might break * update contributing * fix * try 0.63 * fix readme * order of cells * fix env * uncomment lines * try a job * add job test * try miniconda image * try mcr image * -n -> --name for az ml workspace create * fix workspace create * same * isolate rg * fix filename * simply env; increment version * add mlflow * fix enviornment, increment version * switch to .sh * add workspace create * quick fix * fix typo * add asset creation * debugging * fix action * fix * typo * fix typo * workflow edits * fix? * edits * fix data? * Add more commands * try something * remove notebook * update * fix? * try data workaround * fix training script * add azureml-mlflow to env * saving * try sweep job * sweep edits * fix yaml * remove limits * testing * edits * minor edits to basic.yml * add exp name * saving * save; green? * move --web to untested.sh * hmm * add model assets * remove dsstore * saving w/ model create * add endpoints dir * try something for docs * try for docs * try anon data and assets * edit * path edit * fix asset path * move stuff around * try ./ * fix mistake * save * add mount * fix * change model to use run * remove model register * data hack * test * test registered file dataset * saving * testing * whoops * change strategy * fix name * switch rg back * switch creds * switch lightgbm back * fix ws * add file: * remove deepspeed * try tensorflow * remove tensorflow * edits for docs * add verify * unisntall -> remove * -0 * saving * add check_extensions * fix mistake * saving; might break * update manage-assets and test it * whoops * minor edits * local hello world * edits * simplify workflow * try compute job * ? * try compute type * fix hello world * fix? * major refactor * remove line * comments * save * save * order of operations * saving - run sweep job * magic * switch to data - possible typos * fix typos? * ... * try dedenting for sweep * edits * names * sweep hack for now * broke stuff? * test distributed * update query logic * add exp name, :1 to env * wait for job to complete * fix user error * env to version 11 * readme edit * try something * remove redundant stuff * version * try downloading cifar * try job status check * debug bash * fixes * try stuff * add back log upload * fix bash? * copy examples/endpoints from v2-preview * run formatting * add batch endpoint (#427) * hack * move compute create to setup.sh; --name to -n for workspace create * NC6 -> NC12 * try randint sweep * remove randint from sweep; download=False for pytorch-cifar * comment out logs upload from worklow * slim iris lightgbm code * try something for cifar * fix * preprocess -> process * mlflow.pytorch log model for cifar * no mlflow, fix cifar * run formatting * relocate hello world * whoops * cifar epochs 20 -> 2 * try new env for pytorch gpu * switch pytorch to mlflow logging * cleanup cifar code * fix * write to ./model? * edit batch endpoint asset path * force add model.pkl * copy workspace assets * add some schemas, more needed * add more schemas * fix schemas * Updating the schema link to CDN endpoint (#428) * Update batch endpoint test (#429) * update schemas * fix more schemas * simplify * Revert "simplify" This reverts commit 38ec77e0ca9b54051d36699573dea04c0abb583b. * Update batch endpoint sample (#432) * test python/cli refactor (#431) * saving * Update environment.yml (#430) * ? * testing * testing * testing * cli fix * fix * saving * saving * whoops * fix readme * fix readme * whoops * minor readme edit * last readme edits * add setup workspace step * fix cli tests (#433) * test * testing * fixes * save * test * fix how-to-manage-assets * minor edit * fix * fix setup-workspace * fix readme (#434) * edits for docs (#437) * fix readme * remove sweep hack * readme fix * command => sweep in schema * saving * add back setup-workspace.sh * ? * sweep hack * Saving * edit readme asset logic * -A * minor edits for docs (#438) * fix readme * remove sweep hack * readme fix * command => sweep in schema * saving * add back setup-workspace.sh * ? * sweep hack * Saving * edit readme asset logic * -A * >- => | * missed hello world * remove old env * blind copy * yolo * edits * edits * docs edits (#440) * docs edits * save * remove upgrade * fix * reorder * save (#442) * edit training jobs (#443) * saving * minor edit * fix world language? (#444) * change tf horovod env (#445) * change tf horovod env * - * move exp name * standardize cli jobs (#446) * test * standardize training jobs * online endpoints (#435) * online endpoints * run readme.py; manual edit workflows * fix delete command * fix * fix endpoint * rerun readme Co-authored-by: Cody <54814569+lostmygithubaccount@users.noreply.github.com> * quick fix (#448) * run readme (#449) * docs updates (#450) * minor edit for docs (#451) * docs updates * quick fix * undo sweep hack (#452) * edits for docs (#453) * add hello world back * updates * minor edits * Update python-ml-basic-cpu.yml * Add REST examples (#439) * Add scripts without blob cli * Everything working for endpoint except discrimnator issue with online deployment * Working simple job flow * Use default container for data * Rename scripts, make wait_for_completion a function, fix online deployment * Add scoring to endpoint * Change structure of folders * Remove todo * Move rest to cli folder * Switch to lightgbm from xgboost * Fix wait_for_completion * Get AZURE_STORAGE_KEY from az cli * Change MIR region to westus * Remove exports from variables * Fix scoring the endpoint Co-authored-by: Trent McCormick <trmccorm@microsoft.com> * Fix closing tag of sample to match opening tag. (#455) * updates for docs (#456) * switch cli version back (#457) * / * tests for rest how-to (#454) * run readme * save * test rest with azureml-examples * fixes * run readme * Use old way of getting certain variables Co-authored-by: tmccrmck <tjmcc01@gmail.com> * Update how-to-batch-score.sh (#436) Fix batch asset path * force add files (#458) * endpoint script fix (#459) * fix endpoints * fix endpoints * fix endpoint script * fix endpoints * edits for docs (#461) * use * exp name edits * bash edits * save * save * edit * Get storage info from default datastore (#460) * simplify hello world * move RG (#463) * switch credentials (#464) * debug (#465) * debug * fix typo * fjeiwpaol tefjpao * Tweak to canary-flow shellscript (#466) * Fix `test_green` snippet `test_green` snippet was dupe of `green_10pct_traffic`. Switched to syntax described in https://azure.github.io/azureml-v2-preview/_build/html/overview/concepts/endpoints/online-endpoint/scenarios/declarative-flow.html * Results of `python readme.py` This seems erroneous to me, but... * Changed endpoint name to my-new-endpoint per rest of script * fixed typo in var name Co-authored-by: Cody <54814569+lostmygithubaccount@users.noreply.github.com> * Add managed identity shell script for online endpoint (#467) * Online endpoint managed identities tests shell script * Updated online endpoint managed identity shell scripts Co-authored-by: pansav <pansav@microsoft.com> * imperative flow (#469) * .yaml -> .yml (#470) * .yaml -> yml * update python files * edit * Add get deployment logs to rest docs (#473) * Add getLogs API call * Fix typo * minor edit to hello world (#474) * add -example to exp name * edits for docs (#476) * hacks * order of operations * edits * increment model * edits to batch * fix readme * reduce training times to 1 epoch * Update setup.sh * >- -> > (#477) * update CLI version (#480) * update CLI version * try 0.71 * use 1.0.0a2 * Gopalv/tfserving sample (#479) * WIP * adding logic to extract ACR * ran readme.py * local testing works * sed works * parametrized, added local and remote test * adding workflow file * acr login failing * adding sleep after docker run * workflows * Fixed deployment yaml for online endpoint managed identity example (#482) Co-authored-by: pansav <pansav@microsoft.com> * updates to cli-preview (#484) * testing * update dockerfiles * swithc back to conda yml * test * ? * ? * dead snakes * fix job * remove non-python jobs * test * job descriptions * asset descritptions * docs * save * saveing * Fixed online endpoint automation failures issues (#489) * Fixed automation failures issues * Removed try catch block from run method * Fixed declarative yaml * Updated docker image uri to Ubuntu 18.04 * Update system identity flow to use endpoint show to retrieve object id Co-authored-by: pansav <pansav@microsoft.com> * Add how-to-train-autorest with Python SDK (#481) * Add autorest example * Fixes for REST scripts * Removed linux specific code in shell script (#492) * Removed linux specific code in shell script * Added delay after system identity is created and also changed secrets id * Changed CLI version for managed identity tests * Fixed declarative yaml file and upgraded to latest CLI version for managed identity * Changed user identity place holder name * Changed online endpoint schema url in yaml files * Fixed endpoint delete command in declarative flow * Fixed file path in declarative sample Co-authored-by: pansav <pansav@microsoft.com> * Fix how-to-deploy-script (#491) * Fix how-to-deploy-script * Use random versions for environmetns * Don't use headers for async operations * Use jq -r everywhere * Print async operation error * Try changing endpoint * Add mlflow model file (#486) * update batch sample (#493) * Gopalv/triton sample (#490) * WIP * adding logic to extract ACR * ran readme.py * local testing works * sed works * parametrized, added local and remote test * adding workflow file * acr login failing * adding sleep after docker run * workflows * updating yaml and script to exit properly * adding deletion * fixing typos * making base endpoint more generic * try after deletion * renaming tfserving * moved file to correct place * works until MFE error * not tested locally * new CLI drop seems to work * adding triton endpoint * using anonymous environments * adding back placeholder * tfserving fails for some reason * triton works * get different acr * install pyindex * deleting model * fixing model delete * factoring cleanup utility * adding exception handling for acr fail * fixing syntax error * making image url an argument * using public image for tfserving * add back tutorial test, rename * manual modify of recently added workflow * repin black; minor edit * run black-nb * fix typo * test horovod env (#496) * test horovod env * revert to old environment * try v2 * revert * try oss horovod * try tf image * try horoovd image * switch horovod env * 1.0.0a2 -> 1.0.0a4 * update batch sample (#497) * Removed min/max instances and changed auth_mode to key (#498) Co-authored-by: pansav <pansav@microsoft.com> * saving (#499) * minor reqs edits Co-authored-by: Tracy Chen <47156502+tracychms@users.noreply.github.com> Co-authored-by: Harneet Virk <harnvir@microsoft.com> Co-authored-by: Sethu Raman <rsethur@users.noreply.github.com> Co-authored-by: trent <tjmcc01@gmail.com> Co-authored-by: Trent McCormick <trmccorm@microsoft.com> Co-authored-by: vmagelo <51368918+vmagelo@users.noreply.github.com> Co-authored-by: Larry O'Brien <lobrien@knowing.net> Co-authored-by: psavdekar <45885005+psavdekar@users.noreply.github.com> Co-authored-by: pansav <pansav@microsoft.com> Co-authored-by: Gopal Vashishtha <gopalv@microsoft.com>
2021-05-12 04:53:32 +03:00
[Azure/azureml-examples overall contributing guide.](../CONTRIBUTING.md)
## Pull Requests
Pull requests (PRs) to this repo require review and approval by the Azure Machine Learning team to merge. Please follow the pre-defined template and read all relevant sections below.
**Important:** PRs from forks of this repository are likely to fail automated workflows due to access to secrets. PRs from forks will be considered but may experience additional delay for testing.
### Assets
Example asset YAML files should be placed under the `assets` directory in the appropriate subdirectory. The test for assets currently runs `az ml {asset} create`.
### Jobs
The `jobs` directory is structured by scenario then tool then project. The project directory should contain at least one job YAML file and a source code directory named `src`. A docker context or file can also be placed at the root of the project directory for use in the job file, although using prebuilt docker images is preferred.
Jobs should:
- use public cloud data
- use inline environment definitions
- use inline data definitions
- have a good description
- use a pre-existing compute target (defined in `setup.sh`)
- follow the YAML section order `code > command > inputs > environment > compute > experiment_name > description`
### Endpoints
Endpoints are currently only tested through documentation scripts.
refactor for CLI 2.0 preview (#495) * scaffolding * fixes, might break * update contributing * fix * try 0.63 * fix readme * order of cells * fix env * uncomment lines * try a job * add job test * try miniconda image * try mcr image * -n -> --name for az ml workspace create * fix workspace create * same * isolate rg * fix filename * simply env; increment version * add mlflow * fix enviornment, increment version * switch to .sh * add workspace create * quick fix * fix typo * add asset creation * debugging * fix action * fix * typo * fix typo * workflow edits * fix? * edits * fix data? * Add more commands * try something * remove notebook * update * fix? * try data workaround * fix training script * add azureml-mlflow to env * saving * try sweep job * sweep edits * fix yaml * remove limits * testing * edits * minor edits to basic.yml * add exp name * saving * save; green? * move --web to untested.sh * hmm * add model assets * remove dsstore * saving w/ model create * add endpoints dir * try something for docs * try for docs * try anon data and assets * edit * path edit * fix asset path * move stuff around * try ./ * fix mistake * save * add mount * fix * change model to use run * remove model register * data hack * test * test registered file dataset * saving * testing * whoops * change strategy * fix name * switch rg back * switch creds * switch lightgbm back * fix ws * add file: * remove deepspeed * try tensorflow * remove tensorflow * edits for docs * add verify * unisntall -> remove * -0 * saving * add check_extensions * fix mistake * saving; might break * update manage-assets and test it * whoops * minor edits * local hello world * edits * simplify workflow * try compute job * ? * try compute type * fix hello world * fix? * major refactor * remove line * comments * save * save * order of operations * saving - run sweep job * magic * switch to data - possible typos * fix typos? * ... * try dedenting for sweep * edits * names * sweep hack for now * broke stuff? * test distributed * update query logic * add exp name, :1 to env * wait for job to complete * fix user error * env to version 11 * readme edit * try something * remove redundant stuff * version * try downloading cifar * try job status check * debug bash * fixes * try stuff * add back log upload * fix bash? * copy examples/endpoints from v2-preview * run formatting * add batch endpoint (#427) * hack * move compute create to setup.sh; --name to -n for workspace create * NC6 -> NC12 * try randint sweep * remove randint from sweep; download=False for pytorch-cifar * comment out logs upload from worklow * slim iris lightgbm code * try something for cifar * fix * preprocess -> process * mlflow.pytorch log model for cifar * no mlflow, fix cifar * run formatting * relocate hello world * whoops * cifar epochs 20 -> 2 * try new env for pytorch gpu * switch pytorch to mlflow logging * cleanup cifar code * fix * write to ./model? * edit batch endpoint asset path * force add model.pkl * copy workspace assets * add some schemas, more needed * add more schemas * fix schemas * Updating the schema link to CDN endpoint (#428) * Update batch endpoint test (#429) * update schemas * fix more schemas * simplify * Revert "simplify" This reverts commit 38ec77e0ca9b54051d36699573dea04c0abb583b. * Update batch endpoint sample (#432) * test python/cli refactor (#431) * saving * Update environment.yml (#430) * ? * testing * testing * testing * cli fix * fix * saving * saving * whoops * fix readme * fix readme * whoops * minor readme edit * last readme edits * add setup workspace step * fix cli tests (#433) * test * testing * fixes * save * test * fix how-to-manage-assets * minor edit * fix * fix setup-workspace * fix readme (#434) * edits for docs (#437) * fix readme * remove sweep hack * readme fix * command => sweep in schema * saving * add back setup-workspace.sh * ? * sweep hack * Saving * edit readme asset logic * -A * minor edits for docs (#438) * fix readme * remove sweep hack * readme fix * command => sweep in schema * saving * add back setup-workspace.sh * ? * sweep hack * Saving * edit readme asset logic * -A * >- => | * missed hello world * remove old env * blind copy * yolo * edits * edits * docs edits (#440) * docs edits * save * remove upgrade * fix * reorder * save (#442) * edit training jobs (#443) * saving * minor edit * fix world language? (#444) * change tf horovod env (#445) * change tf horovod env * - * move exp name * standardize cli jobs (#446) * test * standardize training jobs * online endpoints (#435) * online endpoints * run readme.py; manual edit workflows * fix delete command * fix * fix endpoint * rerun readme Co-authored-by: Cody <54814569+lostmygithubaccount@users.noreply.github.com> * quick fix (#448) * run readme (#449) * docs updates (#450) * minor edit for docs (#451) * docs updates * quick fix * undo sweep hack (#452) * edits for docs (#453) * add hello world back * updates * minor edits * Update python-ml-basic-cpu.yml * Add REST examples (#439) * Add scripts without blob cli * Everything working for endpoint except discrimnator issue with online deployment * Working simple job flow * Use default container for data * Rename scripts, make wait_for_completion a function, fix online deployment * Add scoring to endpoint * Change structure of folders * Remove todo * Move rest to cli folder * Switch to lightgbm from xgboost * Fix wait_for_completion * Get AZURE_STORAGE_KEY from az cli * Change MIR region to westus * Remove exports from variables * Fix scoring the endpoint Co-authored-by: Trent McCormick <trmccorm@microsoft.com> * Fix closing tag of sample to match opening tag. (#455) * updates for docs (#456) * switch cli version back (#457) * / * tests for rest how-to (#454) * run readme * save * test rest with azureml-examples * fixes * run readme * Use old way of getting certain variables Co-authored-by: tmccrmck <tjmcc01@gmail.com> * Update how-to-batch-score.sh (#436) Fix batch asset path * force add files (#458) * endpoint script fix (#459) * fix endpoints * fix endpoints * fix endpoint script * fix endpoints * edits for docs (#461) * use * exp name edits * bash edits * save * save * edit * Get storage info from default datastore (#460) * simplify hello world * move RG (#463) * switch credentials (#464) * debug (#465) * debug * fix typo * fjeiwpaol tefjpao * Tweak to canary-flow shellscript (#466) * Fix `test_green` snippet `test_green` snippet was dupe of `green_10pct_traffic`. Switched to syntax described in https://azure.github.io/azureml-v2-preview/_build/html/overview/concepts/endpoints/online-endpoint/scenarios/declarative-flow.html * Results of `python readme.py` This seems erroneous to me, but... * Changed endpoint name to my-new-endpoint per rest of script * fixed typo in var name Co-authored-by: Cody <54814569+lostmygithubaccount@users.noreply.github.com> * Add managed identity shell script for online endpoint (#467) * Online endpoint managed identities tests shell script * Updated online endpoint managed identity shell scripts Co-authored-by: pansav <pansav@microsoft.com> * imperative flow (#469) * .yaml -> .yml (#470) * .yaml -> yml * update python files * edit * Add get deployment logs to rest docs (#473) * Add getLogs API call * Fix typo * minor edit to hello world (#474) * add -example to exp name * edits for docs (#476) * hacks * order of operations * edits * increment model * edits to batch * fix readme * reduce training times to 1 epoch * Update setup.sh * >- -> > (#477) * update CLI version (#480) * update CLI version * try 0.71 * use 1.0.0a2 * Gopalv/tfserving sample (#479) * WIP * adding logic to extract ACR * ran readme.py * local testing works * sed works * parametrized, added local and remote test * adding workflow file * acr login failing * adding sleep after docker run * workflows * Fixed deployment yaml for online endpoint managed identity example (#482) Co-authored-by: pansav <pansav@microsoft.com> * updates to cli-preview (#484) * testing * update dockerfiles * swithc back to conda yml * test * ? * ? * dead snakes * fix job * remove non-python jobs * test * job descriptions * asset descritptions * docs * save * saveing * Fixed online endpoint automation failures issues (#489) * Fixed automation failures issues * Removed try catch block from run method * Fixed declarative yaml * Updated docker image uri to Ubuntu 18.04 * Update system identity flow to use endpoint show to retrieve object id Co-authored-by: pansav <pansav@microsoft.com> * Add how-to-train-autorest with Python SDK (#481) * Add autorest example * Fixes for REST scripts * Removed linux specific code in shell script (#492) * Removed linux specific code in shell script * Added delay after system identity is created and also changed secrets id * Changed CLI version for managed identity tests * Fixed declarative yaml file and upgraded to latest CLI version for managed identity * Changed user identity place holder name * Changed online endpoint schema url in yaml files * Fixed endpoint delete command in declarative flow * Fixed file path in declarative sample Co-authored-by: pansav <pansav@microsoft.com> * Fix how-to-deploy-script (#491) * Fix how-to-deploy-script * Use random versions for environmetns * Don't use headers for async operations * Use jq -r everywhere * Print async operation error * Try changing endpoint * Add mlflow model file (#486) * update batch sample (#493) * Gopalv/triton sample (#490) * WIP * adding logic to extract ACR * ran readme.py * local testing works * sed works * parametrized, added local and remote test * adding workflow file * acr login failing * adding sleep after docker run * workflows * updating yaml and script to exit properly * adding deletion * fixing typos * making base endpoint more generic * try after deletion * renaming tfserving * moved file to correct place * works until MFE error * not tested locally * new CLI drop seems to work * adding triton endpoint * using anonymous environments * adding back placeholder * tfserving fails for some reason * triton works * get different acr * install pyindex * deleting model * fixing model delete * factoring cleanup utility * adding exception handling for acr fail * fixing syntax error * making image url an argument * using public image for tfserving * add back tutorial test, rename * manual modify of recently added workflow * repin black; minor edit * run black-nb * fix typo * test horovod env (#496) * test horovod env * revert to old environment * try v2 * revert * try oss horovod * try tf image * try horoovd image * switch horovod env * 1.0.0a2 -> 1.0.0a4 * update batch sample (#497) * Removed min/max instances and changed auth_mode to key (#498) Co-authored-by: pansav <pansav@microsoft.com> * saving (#499) * minor reqs edits Co-authored-by: Tracy Chen <47156502+tracychms@users.noreply.github.com> Co-authored-by: Harneet Virk <harnvir@microsoft.com> Co-authored-by: Sethu Raman <rsethur@users.noreply.github.com> Co-authored-by: trent <tjmcc01@gmail.com> Co-authored-by: Trent McCormick <trmccorm@microsoft.com> Co-authored-by: vmagelo <51368918+vmagelo@users.noreply.github.com> Co-authored-by: Larry O'Brien <lobrien@knowing.net> Co-authored-by: psavdekar <45885005+psavdekar@users.noreply.github.com> Co-authored-by: pansav <pansav@microsoft.com> Co-authored-by: Gopal Vashishtha <gopalv@microsoft.com>
2021-05-12 04:53:32 +03:00
### Scripts
Scripts are bash scripts with the `.sh` extension at the root of the directory. They are often used in the MicrosoftDocs/azure-docs repository as the source for code snippets.
refactor for CLI 2.0 preview (#495) * scaffolding * fixes, might break * update contributing * fix * try 0.63 * fix readme * order of cells * fix env * uncomment lines * try a job * add job test * try miniconda image * try mcr image * -n -> --name for az ml workspace create * fix workspace create * same * isolate rg * fix filename * simply env; increment version * add mlflow * fix enviornment, increment version * switch to .sh * add workspace create * quick fix * fix typo * add asset creation * debugging * fix action * fix * typo * fix typo * workflow edits * fix? * edits * fix data? * Add more commands * try something * remove notebook * update * fix? * try data workaround * fix training script * add azureml-mlflow to env * saving * try sweep job * sweep edits * fix yaml * remove limits * testing * edits * minor edits to basic.yml * add exp name * saving * save; green? * move --web to untested.sh * hmm * add model assets * remove dsstore * saving w/ model create * add endpoints dir * try something for docs * try for docs * try anon data and assets * edit * path edit * fix asset path * move stuff around * try ./ * fix mistake * save * add mount * fix * change model to use run * remove model register * data hack * test * test registered file dataset * saving * testing * whoops * change strategy * fix name * switch rg back * switch creds * switch lightgbm back * fix ws * add file: * remove deepspeed * try tensorflow * remove tensorflow * edits for docs * add verify * unisntall -> remove * -0 * saving * add check_extensions * fix mistake * saving; might break * update manage-assets and test it * whoops * minor edits * local hello world * edits * simplify workflow * try compute job * ? * try compute type * fix hello world * fix? * major refactor * remove line * comments * save * save * order of operations * saving - run sweep job * magic * switch to data - possible typos * fix typos? * ... * try dedenting for sweep * edits * names * sweep hack for now * broke stuff? * test distributed * update query logic * add exp name, :1 to env * wait for job to complete * fix user error * env to version 11 * readme edit * try something * remove redundant stuff * version * try downloading cifar * try job status check * debug bash * fixes * try stuff * add back log upload * fix bash? * copy examples/endpoints from v2-preview * run formatting * add batch endpoint (#427) * hack * move compute create to setup.sh; --name to -n for workspace create * NC6 -> NC12 * try randint sweep * remove randint from sweep; download=False for pytorch-cifar * comment out logs upload from worklow * slim iris lightgbm code * try something for cifar * fix * preprocess -> process * mlflow.pytorch log model for cifar * no mlflow, fix cifar * run formatting * relocate hello world * whoops * cifar epochs 20 -> 2 * try new env for pytorch gpu * switch pytorch to mlflow logging * cleanup cifar code * fix * write to ./model? * edit batch endpoint asset path * force add model.pkl * copy workspace assets * add some schemas, more needed * add more schemas * fix schemas * Updating the schema link to CDN endpoint (#428) * Update batch endpoint test (#429) * update schemas * fix more schemas * simplify * Revert "simplify" This reverts commit 38ec77e0ca9b54051d36699573dea04c0abb583b. * Update batch endpoint sample (#432) * test python/cli refactor (#431) * saving * Update environment.yml (#430) * ? * testing * testing * testing * cli fix * fix * saving * saving * whoops * fix readme * fix readme * whoops * minor readme edit * last readme edits * add setup workspace step * fix cli tests (#433) * test * testing * fixes * save * test * fix how-to-manage-assets * minor edit * fix * fix setup-workspace * fix readme (#434) * edits for docs (#437) * fix readme * remove sweep hack * readme fix * command => sweep in schema * saving * add back setup-workspace.sh * ? * sweep hack * Saving * edit readme asset logic * -A * minor edits for docs (#438) * fix readme * remove sweep hack * readme fix * command => sweep in schema * saving * add back setup-workspace.sh * ? * sweep hack * Saving * edit readme asset logic * -A * >- => | * missed hello world * remove old env * blind copy * yolo * edits * edits * docs edits (#440) * docs edits * save * remove upgrade * fix * reorder * save (#442) * edit training jobs (#443) * saving * minor edit * fix world language? (#444) * change tf horovod env (#445) * change tf horovod env * - * move exp name * standardize cli jobs (#446) * test * standardize training jobs * online endpoints (#435) * online endpoints * run readme.py; manual edit workflows * fix delete command * fix * fix endpoint * rerun readme Co-authored-by: Cody <54814569+lostmygithubaccount@users.noreply.github.com> * quick fix (#448) * run readme (#449) * docs updates (#450) * minor edit for docs (#451) * docs updates * quick fix * undo sweep hack (#452) * edits for docs (#453) * add hello world back * updates * minor edits * Update python-ml-basic-cpu.yml * Add REST examples (#439) * Add scripts without blob cli * Everything working for endpoint except discrimnator issue with online deployment * Working simple job flow * Use default container for data * Rename scripts, make wait_for_completion a function, fix online deployment * Add scoring to endpoint * Change structure of folders * Remove todo * Move rest to cli folder * Switch to lightgbm from xgboost * Fix wait_for_completion * Get AZURE_STORAGE_KEY from az cli * Change MIR region to westus * Remove exports from variables * Fix scoring the endpoint Co-authored-by: Trent McCormick <trmccorm@microsoft.com> * Fix closing tag of sample to match opening tag. (#455) * updates for docs (#456) * switch cli version back (#457) * / * tests for rest how-to (#454) * run readme * save * test rest with azureml-examples * fixes * run readme * Use old way of getting certain variables Co-authored-by: tmccrmck <tjmcc01@gmail.com> * Update how-to-batch-score.sh (#436) Fix batch asset path * force add files (#458) * endpoint script fix (#459) * fix endpoints * fix endpoints * fix endpoint script * fix endpoints * edits for docs (#461) * use * exp name edits * bash edits * save * save * edit * Get storage info from default datastore (#460) * simplify hello world * move RG (#463) * switch credentials (#464) * debug (#465) * debug * fix typo * fjeiwpaol tefjpao * Tweak to canary-flow shellscript (#466) * Fix `test_green` snippet `test_green` snippet was dupe of `green_10pct_traffic`. Switched to syntax described in https://azure.github.io/azureml-v2-preview/_build/html/overview/concepts/endpoints/online-endpoint/scenarios/declarative-flow.html * Results of `python readme.py` This seems erroneous to me, but... * Changed endpoint name to my-new-endpoint per rest of script * fixed typo in var name Co-authored-by: Cody <54814569+lostmygithubaccount@users.noreply.github.com> * Add managed identity shell script for online endpoint (#467) * Online endpoint managed identities tests shell script * Updated online endpoint managed identity shell scripts Co-authored-by: pansav <pansav@microsoft.com> * imperative flow (#469) * .yaml -> .yml (#470) * .yaml -> yml * update python files * edit * Add get deployment logs to rest docs (#473) * Add getLogs API call * Fix typo * minor edit to hello world (#474) * add -example to exp name * edits for docs (#476) * hacks * order of operations * edits * increment model * edits to batch * fix readme * reduce training times to 1 epoch * Update setup.sh * >- -> > (#477) * update CLI version (#480) * update CLI version * try 0.71 * use 1.0.0a2 * Gopalv/tfserving sample (#479) * WIP * adding logic to extract ACR * ran readme.py * local testing works * sed works * parametrized, added local and remote test * adding workflow file * acr login failing * adding sleep after docker run * workflows * Fixed deployment yaml for online endpoint managed identity example (#482) Co-authored-by: pansav <pansav@microsoft.com> * updates to cli-preview (#484) * testing * update dockerfiles * swithc back to conda yml * test * ? * ? * dead snakes * fix job * remove non-python jobs * test * job descriptions * asset descritptions * docs * save * saveing * Fixed online endpoint automation failures issues (#489) * Fixed automation failures issues * Removed try catch block from run method * Fixed declarative yaml * Updated docker image uri to Ubuntu 18.04 * Update system identity flow to use endpoint show to retrieve object id Co-authored-by: pansav <pansav@microsoft.com> * Add how-to-train-autorest with Python SDK (#481) * Add autorest example * Fixes for REST scripts * Removed linux specific code in shell script (#492) * Removed linux specific code in shell script * Added delay after system identity is created and also changed secrets id * Changed CLI version for managed identity tests * Fixed declarative yaml file and upgraded to latest CLI version for managed identity * Changed user identity place holder name * Changed online endpoint schema url in yaml files * Fixed endpoint delete command in declarative flow * Fixed file path in declarative sample Co-authored-by: pansav <pansav@microsoft.com> * Fix how-to-deploy-script (#491) * Fix how-to-deploy-script * Use random versions for environmetns * Don't use headers for async operations * Use jq -r everywhere * Print async operation error * Try changing endpoint * Add mlflow model file (#486) * update batch sample (#493) * Gopalv/triton sample (#490) * WIP * adding logic to extract ACR * ran readme.py * local testing works * sed works * parametrized, added local and remote test * adding workflow file * acr login failing * adding sleep after docker run * workflows * updating yaml and script to exit properly * adding deletion * fixing typos * making base endpoint more generic * try after deletion * renaming tfserving * moved file to correct place * works until MFE error * not tested locally * new CLI drop seems to work * adding triton endpoint * using anonymous environments * adding back placeholder * tfserving fails for some reason * triton works * get different acr * install pyindex * deleting model * fixing model delete * factoring cleanup utility * adding exception handling for acr fail * fixing syntax error * making image url an argument * using public image for tfserving * add back tutorial test, rename * manual modify of recently added workflow * repin black; minor edit * run black-nb * fix typo * test horovod env (#496) * test horovod env * revert to old environment * try v2 * revert * try oss horovod * try tf image * try horoovd image * switch horovod env * 1.0.0a2 -> 1.0.0a4 * update batch sample (#497) * Removed min/max instances and changed auth_mode to key (#498) Co-authored-by: pansav <pansav@microsoft.com> * saving (#499) * minor reqs edits Co-authored-by: Tracy Chen <47156502+tracychms@users.noreply.github.com> Co-authored-by: Harneet Virk <harnvir@microsoft.com> Co-authored-by: Sethu Raman <rsethur@users.noreply.github.com> Co-authored-by: trent <tjmcc01@gmail.com> Co-authored-by: Trent McCormick <trmccorm@microsoft.com> Co-authored-by: vmagelo <51368918+vmagelo@users.noreply.github.com> Co-authored-by: Larry O'Brien <lobrien@knowing.net> Co-authored-by: psavdekar <45885005+psavdekar@users.noreply.github.com> Co-authored-by: pansav <pansav@microsoft.com> Co-authored-by: Gopal Vashishtha <gopalv@microsoft.com>
2021-05-12 04:53:32 +03:00
update examples for CLI (v2) public preview refresh (#890) * update setup to use preview CLI * Create setup.sh * update schemas to preview * Update setup.sh * Update setup.sh * Update setup.sh * Update setup.sh * Update setup.sh * add environment examples (#736) * Create docker image example * Rename docker image example to docker_image.yml * Create Dockerfile * Create docker_context.yml * refactor and updates * update creds, readme, workflows * fix * remove workflow * fix mistake * fix Co-authored-by: Cody <54814569+lostmygithubaccount@users.noreply.github.com> * Update docker-context.yml (#739) * Update setup.sh * Update setup.sh * Update setup.sh * Update setup.sh * major jobs refactor; add pipeline job (#745) * add basic and nyc taxi pipelines samples under jobs * major refactor * minor edit * minor edit * save * remove sweep dir * update inputs outputs and compute for cli 0.7 * update component yamls nyc taxi example Co-authored-by: Sharmeelee Bijlani <shbijlan@microsoft.com> * Create simple-compute-cluster.yml (#748) * update compute cluster examples (#752) * update compute examples * shorten name * low_priorty -> lowpriority * add locaiton example * add compute-instance * remove instances from testing * refactor jobs; add simple pipeline examples (#753) * refactor jobs * save * pipeline edits * switch to canary * switch to eastus * fully test * switch back to master * reduce python-sdk testing freq; cli and setup cleanup (#759) * repo health * rerun readme * environment example updates (#760) * remove version pinning; add version pinning * edits * Update setup.sh * add workflow_dispatch: to cli workflows (#761) * touch setup.sh * add workflow_dispatch: to cli jobs * Update setup.sh * Update setup.sh * fix low priority compute example (#762) * Update setup.sh * public preview refresh changes (#750) * fixes * changes * revert setup.sh changes * latest cli * cli changes * update dir structure * update name * update cli version * cli version * cli version * cleanup * update comments Co-authored-by: Cody <54814569+lostmygithubaccount@users.noreply.github.com> * Update setup.sh * drop _job (#772) * Add files via upload * fix validation scripts for user specified identity and system assigned identity (#747) * fix sai + uai scripts * remove set x * fix workspace name * fix arg * fix rest * add sku * change credentials to mir * touch * add check * add extra line ending * add object info to role assignment * forgotten vars * fix role assignment * fix other role assignments * use latest version of sdk * fix arm calls * remove hardcoded vals * fix hardcoded values * replace endpoint name with var * add euap location * remove pined version of cli * fix sai role assignment * update credential logic in readme for uai, sai * format code * fixed comments * cleaned up * cleanup * revert * fix tags * fix uai Co-authored-by: max <public.max@outlook.com> Co-authored-by: Cody <54814569+lostmygithubaccount@users.noreply.github.com> * clean up nyc taxi pipeline (#773) * update sweep timeout (#778) * pin badges to main branch (#779) * pin badges to main branch * save * switch testing to master * manually update schemas * Update setup.sh * Update PyTorch example (#791) * Update setup.sh * add cifar-10 pipeline example (#794) * save * save - training updates * remove cifar from untested jobs * save * save * switch setup to eastus * set CR to false * fix some jobs * update readme * save * save * fix setup * run readme * add data copy script * edit * edit * save * save * fix * save * edit cifar loc * fix * fix * Update setup.sh * batch GA samples (#722) * batch score GA samples * update batch * update batch * update batch sample * update anon env * move non mlflow up * update test flow * clean up batch sample * update batch sample * update schema ref * rename yml files * update yml file name * minor update * Update setup.sh * Update workspace CLI examples (#767) * Update workspace CLI examples * updating examples per PR feedback * property ordering and update app insights attr fix * include example on adding private link endpoint * include private link example * simplify ple example * disabled in capital and consistent resource suffix * consisten resource names * save * fix * add ws delete * fixes * fix * fix * remove workspace script Co-authored-by: Cody <54814569+lostmygithubaccount@users.noreply.github.com> * add diabetes data * fix * fix * save * Fixed typo in snippet (#796) * pipelines-with-components basic examples (#777) * pipelines-with-components * data samples * tf hello world * create components and dataset in repo setup * pytorch hello world * update readme; format code * update create-datasets.sh * fix readme; remove unused job; run readme * add missing files * setup updates Co-authored-by: Cody <54814569+lostmygithubaccount@users.noreply.github.com> * create variables (#799) * local endpoint test script (#804) * script for local endpoint * local endpoint * Added Dataset and Datastore YAML samples. (#800) * Added Dataset and Datastore YAML samples. * Updated README files. * renames and minor updates * Updated Dataset samples README. Co-authored-by: Cody <54814569+lostmygithubaccount@users.noreply.github.com> * standardization and fixes (#805) * update schemas * update assets * update resources * update setup-repo * save * udpate assets * update run logic (#806) * update schemas * update assets * update resources * update setup-repo * save * udpate assets * update run logic * omit run-job.sh * print job uri * remove workflow * edit run-job.sh * update run-job.sh * update run-job.sh * fix cifar pipeline (#809) * Update setup.sh * ; => : (#812) * model-dir => model_dir (#813) * update run-job.sh * mount => ro_mount (#814) * fix nyc-taxi pipeline example (#815) * ever feel stupid? I feel stupid * fix 4d_dataset_input (#816) * clean up batch endpoints (#817) * switch RG (#818) * Update setup.sh * format code * update jobs and things (#822) * save * save * save * save * save * save * save * save * format code * remove cli/jobs/basics/data * save * save * save * update * fix * minor edit * edit * switch to eastus * format code * Add functional Batch Endpoint REST sample (#820) * add batch score rest sample * add git workflow * update REST sample * update sample * Functional October batch REST sample -- score from Dataset failing on request * Clean up script * run readme * Fix some quoting issues * reset location * Fix modelUri * Fail early if job submit failed * Reset readme.py * Fix location to be based on workspace * Add other terminal state Co-authored-by: Tracy Chen <47156502+tracychms@users.noreply.github.com> Co-authored-by: Cody <54814569+lostmygithubaccount@users.noreply.github.com> * minor edits * save * update * job updates (#824) * save * updates * minor updates * minor update * fix cifar dataset * Update hello-iris-file.yml * fixes (#827) * fixes * fix * fix * add docker context job example for sklearn (#828) * add docker context job example for sklearn * run readme * change image * Add missing section opener (#829) Missing opening comment for create_environment * minor edit * remove arc examples (#830) * save * add hello-code * autoscale (#832) * fix unique output (#831) * fix unique output * fix unique output * beautify json * update REST invoke * fix typo * remove delete prompt * unique output folder * add deployment delete * consistency edits (#834) * --set experiment_name in train.sh (#835) * fix instance (#836) * minor consistency edits (#837) * removing aks sample from cli-preview (#844) * Update torchserve BYOC sample (#845) * add notebooks via papermill job examples (#847) * add notebooks via papermill job examples * output notebook to ./outputs/ * fix tag (#848) * byoc tfserving sample (#853) * byoc tfserving sample * deployment name fix * r-byoc-update (#854) * Update setup.sh * update * update * update vm attach example (#855) * Revert "update vm attach example" (#856) This reverts commit f3a7efe4fa0e919582aaf5c4ae206368dbc1470b. * add env version for component examples (#858) * fix typo in directory; rw-mount to rw_mount (#859) * fix spark example (#857) * fix spark example * save * try pin pip * ? * try older pip * pin image * format code * update rest sample (#861) * update rest sample * fix format * fix typo * fix output path * update outputdataset to output * update output * move dask to named output * add tag for wait (#866) * Fixing closing region tag (#867) read_condafile section tags were not correctly closed. * swap install steps in setup.sh * edit comment * run black * remove CIs from refresh testing (#873) * add defaults list to misc script (#874) * https => wasbs for dask folder example (#875) * https => wasbs for dask folder example * test moving container * Adding MLflow sample for online endpoints (#872) * Adding MLflow sample for online endpoints * run readme.py * run black * Revert "run black" This reverts commit b649bb564df6eecfe3118d6ab0d74268c6c680f3. * updating sh file * Adding MLflow model deployment sample * Updating files to be consistent * updating .sh file * updating deployment YAML and .sh file * adding debug flag for model delete * updating YAML and .sh script * updating .sh script Co-authored-by: Cody <54814569+lostmygithubaccount@users.noreply.github.com> * Update triton samples to ncd - single model (#876) * Update triton samples to ncd - single model * minor spelling error * addressing build failures -1 * addressing build failures -2 * addressing missing dependencies * adding missing dependency 2 * update pip install tritonclient[http] * addressing build failures * minor update to tags (#877) * remove extra triton examples (#881) * remove flaml notebook action * add ssh settings to vm attach (#883) * remove .schemas; consistency edit to scripts (#887) * remove .schemas * consistency edit * remove unused workflow * switch cli install to public * remove broken job (#892) * fix merge issues * fix job logic Co-authored-by: Saachi Gopal <sg932@cornell.edu> Co-authored-by: Sharmeelee Bijlani <shbijlan@microsoft.com> Co-authored-by: vijetajo <40418529+vijetajo@users.noreply.github.com> Co-authored-by: Sethu Raman <rsethur@users.noreply.github.com> Co-authored-by: Bala P V <33712765+balapv@users.noreply.github.com> Co-authored-by: max markin <maxormo@gmail.com> Co-authored-by: max <public.max@outlook.com> Co-authored-by: Paul Shealy <paulsh@microsoft.com> Co-authored-by: Tracy Chen <47156502+tracychms@users.noreply.github.com> Co-authored-by: Dennis Eikelenboom <mail@denniseik.nl> Co-authored-by: Larry O'Brien <lobrien@knowing.net> Co-authored-by: Manoj Bableshwar <manoj.bableshwar@outlook.com> Co-authored-by: Yogi Pandey <20666257+ynpandey@users.noreply.github.com> Co-authored-by: Christopher Pirillo <pirillo.chris@gmail.com> Co-authored-by: Larry Franks <larry.franks@microsoft.com> Co-authored-by: Shivani Santosh Sambare <61255242+shivanissambare@users.noreply.github.com> Co-authored-by: Daniel Schneider <daniel.schneider@microsoft.com>
2021-11-04 17:46:46 +03:00
### Important: Suppress printing confidential data/secrets in your scripts
Note that all the shell scripts are executed with `bash -x script.sh`. This will print output of each line in the workflow output, which will be useful for debugging. However if you have sensitive information, that will get printed too. For e.g.:
```bash
# get endpoint access key using CLI command - this is confidential data
ENDPOINT_CREDENTIALS=$(az ml online-endpoint get-credentials -n $ENDPOINT_NAME -o tsv --query primaryKey)
```
in this case the output in the workflow run will have endpoint key printed.
To disable printing the output for selected lines by enclosing them within `set +x` and `set -x`. Example
```bash
set +x
ENDPOINT_CREDENTIALS=$(az ml online-endpoint get-credentials -n $ENDPOINT_NAME -o tsv --query primaryKey)
set -x
```
You can also add the environment variable as a [github secret](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository) to ensure it is not printed elsewhere.