edit github workflow naming conventions (#364)

* edit github workflow naming conventions

* edits to contributing

* a few more minor edits while we're here

* prevent bug
This commit is contained in:
Cody 2021-02-08 15:44:44 -08:00 коммит произвёл GitHub
Родитель 088ce2c552
Коммит ba95da855d
14 изменённых файлов: 51 добавлений и 53 удалений

Просмотреть файл

@ -1,4 +1,4 @@
name: notebook-tll
name: notebook-train-lightgbm-local
on:
schedule:
- cron: "0 0/2 * * *"
@ -7,7 +7,7 @@ on:
- main
paths:
- notebooks/train-lightgbm-local.ipynb
- .github/workflows/notebook-tll.yml
- .github/workflows/notebook-train-lightgbm-local.yml
jobs:
build:
runs-on: ubuntu-latest

Просмотреть файл

@ -1,4 +1,4 @@
name: tutorial-ai
name: tutorial-an-introduction
on:
schedule:
- cron: "0 0/1 * * *"
@ -7,7 +7,7 @@ on:
- main
paths:
- tutorials/an-introduction/**
- .github/workflows/tutorial-ai.yml
- .github/workflows/tutorial-an-introduction.yml
jobs:
build:
runs-on: ubuntu-latest

Просмотреть файл

@ -1,4 +1,4 @@
name: tutorial-awp
name: tutorial-automl-with-pycaret
on:
schedule:
- cron: "0 0/2 * * *"
@ -7,7 +7,7 @@ on:
- main
paths:
- tutorials/automl-with-pycaret/**
- .github/workflows/tutorial-awp.yml
- .github/workflows/tutorial-automl-with-pycaret.yml
jobs:
build:
runs-on: ubuntu-latest

Просмотреть файл

@ -1,4 +1,4 @@
name: tutorial-de
name: tutorial-deploy-edge
on:
schedule:
- cron: "0 0/2 * * *"
@ -7,7 +7,7 @@ on:
- main
paths:
- tutorials/deploy-edge/**
- .github/workflows/tutorial-de.yml
- .github/workflows/tutorial-deploy-edge.yml
jobs:
build:
runs-on: ubuntu-latest

Просмотреть файл

@ -1,4 +1,4 @@
name: tutorial-dt
name: tutorial-deploy-triton
on:
schedule:
- cron: "0 */2 * * *"
@ -7,7 +7,7 @@ on:
- main
paths:
- tutorials/deploy-triton/**
- .github/workflows/tutorial-dt.yml
- .github/workflows/tutorial-deploy-triton.yml
jobs:
build:
runs-on: ubuntu-latest

Просмотреть файл

@ -1,4 +1,4 @@
name: tutorial-ud
name: tutorial-using-dask
on:
schedule:
- cron: "0 */2 * * *"
@ -7,7 +7,7 @@ on:
- main
paths:
- tutorials/using-dask/**
- .github/workflows/tutorial-ud.yml
- .github/workflows/tutorial-using-dask.yml
jobs:
build:
runs-on: ubuntu-latest

Просмотреть файл

@ -1,4 +1,4 @@
name: tutorial-upl
name: tutorial-using-pytorch-lightning
on:
schedule:
- cron: "0 */2 * * *"
@ -7,7 +7,7 @@ on:
- main
paths:
- tutorials/using-pytorch-lightning/**
- .github/workflows/tutorial-upl.yml
- .github/workflows/tutorial-using-pytorch-lightning.yml
jobs:
build:
runs-on: ubuntu-latest

Просмотреть файл

@ -1,4 +1,4 @@
name: tutorial-ur
name: tutorial-using-rapids
on:
schedule:
- cron: "0 */2 * * *"
@ -7,7 +7,7 @@ on:
- main
paths:
- tutorials/using-rapids/**
- .github/workflows/tutorial-ur.yml
- .github/workflows/tutorial-using-rapids.yml
jobs:
build:
runs-on: ubuntu-latest

Просмотреть файл

@ -1,4 +1,4 @@
name: tutorial-ux
name: tutorial-using-xgboost
on:
schedule:
- cron: "0 */2 * * *"
@ -7,7 +7,7 @@ on:
- main
paths:
- tutorials/using-xgboost/**
- .github/workflows/tutorial-ux.yml
- .github/workflows/tutorial-using-xgboost.yml
jobs:
build:
runs-on: ubuntu-latest

Просмотреть файл

@ -26,14 +26,15 @@ This repo is an opinionated set of examples using a subset of Azure Machine Lear
## Issues
All forms of feedback are welcome through issues - please follow the pre-defined templates where applicable.
All forms of feedback are welcome through [issues](https://github.com/Azure/azureml-examples/issues) - please follow the pre-defined templates where applicable.
*Note:* [Discussions](https://github.com/Azure/azureml-examples/discussions) are new to GitHub, feel free to start one!
## Pull Requests
Pull requests (PRs) to this repo require review and approval by the Azure Machine Learning team to merge. Please follow the pre-defined template and read all relevant sections below.
> **Important:**
> PRs from forks of this repository are likely to fail automated workflows due to access to secrets. PRs from forks will be considered but may experience additional delay for testing.
**Important:** PRs from forks of this repository are likely to fail automated workflows due to access to secrets. PRs from forks will be considered but may experience additional delay for testing.
### General rules
@ -44,13 +45,14 @@ Pull requests (PRs) to this repo require review and approval by the Azure Machin
### Miscellaneous
- to modify `README.md`, you need to modify `readme.py` and accompanying markdown files
- the tables in the `README.md` are auto-generated, including description, via other files
- to modify `README.md`, you need to modify `readme.py` and accompanying markdown files other files (`prefix.md` and `suffix.md`)
- you probably should not modify `README.md` or accompany markdown files
- develop on a branch, not a fork, for workflows to run properly
- use an existing environment where possible
- use an existing dataset where possible
- don't register environments
- don't create compute targets
- don't register datasets
- don't modify `requirements.txt`
- you probably shouldn't modify any files in the root of the repo
- you can `!pip install --upgrade packages` as needed in notebooks
@ -62,7 +64,7 @@ If modifying existing examples, before a PR:
- run `python readme.py` from the root of the repo
- this will generate the `README.md` file
- this will generate github workflow files
- this will generate github workflow files (for workflows and notebooks)
- this will format Python code and notebooks
### Enforced naming
@ -73,7 +75,7 @@ Enforced naming includes:
- directories under `tutorials` must be words separated by hyphens
- directories under `workflows` must be one of [`train`, `deploy`, `score`, `dataprep`] - directories under are organized by ML tool
- job definition file(s) under `workflows` must contain `job` in the name
- tutorial workflows (and workflow files) use the naming convention `tutorial-*initials*`, where *initials* is the initials of the words
- tutorial workflows (and workflow files) use the naming convention `tutorial-*name*`, where *name* is the directory name
- `experiment_name` = "logical-words-example|tutorial" e.g. "hello-world-tutorial"
- `compute_name` = "compute-defined-in-setup-workspace.py" e.g. "gpu-K80-2"
@ -146,7 +148,6 @@ Checklist:
- [ ] use an existing directory or add a new scenario and/or ML tool directory
- [ ] add job definition file(s) under this directory with `job` in the name
- [ ] add user code, preserving any licensing information, under a `src` dir specific to the workflow
- [ ] add any required environment files under a `envs` dir, shared commonly across workflows
- [ ] run `python readme.py`
- [ ] test
- [ ] submit PR, which will run the relevant workflow(s)
@ -171,7 +172,7 @@ Checklist:
- [ ] add the tutorial directory under `tutorials/`, following naming conventions
- [ ] add tutorial files, which are usually notebooks and may be ordered
- [ ] add `README.md` in the tutorial directory with a description (see other tutorials for format)
- [ ] add `run-tutorial-initials`, where *initials* are the initials of the description directory (see other tutorial workflows)
- [ ] add `tutorial-*name*`, where *name* is the name of the directory (see other tutorial workflows)
- [ ] run `python readme.py`
- [ ] test
- [ ] submit PR, which will run your tutorial if setup properly

Просмотреть файл

@ -48,7 +48,7 @@ A lightweight template repository for automating the ML lifecycle can be found [
|`.cloud`|cloud templates|
|`.github`|GitHub specific files like Actions workflow yaml definitions and issue templates|
|`notebooks`|interactive jupyter notebooks for iterative ML development|
|`tutorials`|self-contained directories of end-to-end tutorials|
|`tutorials`|self-contained directories of tutorials|
|`workflows`|self-contained directories of job to be run, organized by scenario then tool then project|
## Examples
@ -57,20 +57,20 @@ A lightweight template repository for automating the ML lifecycle can be found [
path|status|notebooks|description
-|-|-|-
[an-introduction](tutorials/an-introduction)|[![an-introduction](https://github.com/Azure/azureml-examples/workflows/tutorial-ai/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-ai)|[1.hello-world.ipynb](tutorials/an-introduction/1.hello-world.ipynb)<br>[2.pytorch-model.ipynb](tutorials/an-introduction/2.pytorch-model.ipynb)<br>[3.pytorch-model-cloud-data.ipynb](tutorials/an-introduction/3.pytorch-model-cloud-data.ipynb)|learn the basics of Azure Machine Learning
[automl-with-pycaret](tutorials/automl-with-pycaret)|[![automl-with-pycaret](https://github.com/Azure/azureml-examples/workflows/tutorial-awp/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-awp)|[1.classification.ipynb](tutorials/automl-with-pycaret/1.classification.ipynb)|learn how to automate ML with [PyCaret](https://github.com/pycaret/pycaret)
[deploy-edge](tutorials/deploy-edge)|[![deploy-edge](https://github.com/Azure/azureml-examples/workflows/tutorial-de/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-de)|[ase-gpu.ipynb](tutorials/deploy-edge/ase-gpu.ipynb)|learn how to use Edge device for model deployment and scoring
[deploy-triton](tutorials/deploy-triton)|[![deploy-triton](https://github.com/Azure/azureml-examples/workflows/tutorial-dt/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-dt)|[1.densenet-local.ipynb](tutorials/deploy-triton/1.densenet-local.ipynb)<br>[2.bidaf-aks-v100.ipynb](tutorials/deploy-triton/2.bidaf-aks-v100.ipynb)|learn how to efficiently deploy to GPUs using [triton inference server](https://github.com/triton-inference-server/server)
[using-dask](tutorials/using-dask)|[![using-dask](https://github.com/Azure/azureml-examples/workflows/tutorial-ud/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-ud)|[1.intro-to-dask.ipynb](tutorials/using-dask/1.intro-to-dask.ipynb)<br>[2.dask-cloudprovider.ipynb](tutorials/using-dask/2.dask-cloudprovider.ipynb)|learn how to read from cloud data and scale PyData tools (Numpy, Pandas, Scikit-Learn, etc.) with [Dask](https://dask.org)
[using-pytorch-lightning](tutorials/using-pytorch-lightning)|[![using-pytorch-lightning](https://github.com/Azure/azureml-examples/workflows/tutorial-upl/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-upl)|[1.train-single-node.ipynb](tutorials/using-pytorch-lightning/1.train-single-node.ipynb)<br>[2.log-with-tensorboard.ipynb](tutorials/using-pytorch-lightning/2.log-with-tensorboard.ipynb)<br>[3.log-with-mlflow.ipynb](tutorials/using-pytorch-lightning/3.log-with-mlflow.ipynb)<br>[4.train-multi-node-ddp.ipynb](tutorials/using-pytorch-lightning/4.train-multi-node-ddp.ipynb)|learn how to train and log metrics with [PyTorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning)
[using-rapids](tutorials/using-rapids)|[![using-rapids](https://github.com/Azure/azureml-examples/workflows/tutorial-ur/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-ur)|[1.train-and-hpo.ipynb](tutorials/using-rapids/1.train-and-hpo.ipynb)<br>[2.train-multi-gpu.ipynb](tutorials/using-rapids/2.train-multi-gpu.ipynb)|learn how to accelerate PyData tools (numpy, pandas, scikit-learn, etc) on NVIDIA GPUs with [rapids](https://github.com/rapidsai)
[using-xgboost](tutorials/using-xgboost)|[![using-xgboost](https://github.com/Azure/azureml-examples/workflows/tutorial-ux/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-ux)|[1.local-eda.ipynb](tutorials/using-xgboost/1.local-eda.ipynb)<br>[2.distributed-cpu.ipynb](tutorials/using-xgboost/2.distributed-cpu.ipynb)|learn how to use [XGBoost](https://github.com/dmlc/xgboost) on Azure
[an-introduction](tutorials/an-introduction)|[![an-introduction](https://github.com/Azure/azureml-examples/workflows/tutorial-an-introduction/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-an-introduction)|[1.hello-world.ipynb](tutorials/an-introduction/1.hello-world.ipynb)<br>[2.pytorch-model.ipynb](tutorials/an-introduction/2.pytorch-model.ipynb)<br>[3.pytorch-model-cloud-data.ipynb](tutorials/an-introduction/3.pytorch-model-cloud-data.ipynb)|learn the basics of Azure Machine Learning
[automl-with-pycaret](tutorials/automl-with-pycaret)|[![automl-with-pycaret](https://github.com/Azure/azureml-examples/workflows/tutorial-automl-with-pycaret/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-automl-with-pycaret)|[1.classification.ipynb](tutorials/automl-with-pycaret/1.classification.ipynb)|learn how to automate ML with [PyCaret](https://github.com/pycaret/pycaret)
[deploy-edge](tutorials/deploy-edge)|[![deploy-edge](https://github.com/Azure/azureml-examples/workflows/tutorial-deploy-edge/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-deploy-edge)|[ase-gpu.ipynb](tutorials/deploy-edge/ase-gpu.ipynb)|learn how to use Edge device for model deployment and scoring
[deploy-triton](tutorials/deploy-triton)|[![deploy-triton](https://github.com/Azure/azureml-examples/workflows/tutorial-deploy-triton/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-deploy-triton)|[1.densenet-local.ipynb](tutorials/deploy-triton/1.densenet-local.ipynb)<br>[2.bidaf-aks-v100.ipynb](tutorials/deploy-triton/2.bidaf-aks-v100.ipynb)|learn how to efficiently deploy to GPUs using [triton inference server](https://github.com/triton-inference-server/server)
[using-dask](tutorials/using-dask)|[![using-dask](https://github.com/Azure/azureml-examples/workflows/tutorial-using-dask/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-using-dask)|[1.intro-to-dask.ipynb](tutorials/using-dask/1.intro-to-dask.ipynb)<br>[2.dask-cloudprovider.ipynb](tutorials/using-dask/2.dask-cloudprovider.ipynb)|learn how to read from cloud data and scale PyData tools (Numpy, Pandas, Scikit-Learn, etc.) with [Dask](https://dask.org)
[using-pytorch-lightning](tutorials/using-pytorch-lightning)|[![using-pytorch-lightning](https://github.com/Azure/azureml-examples/workflows/tutorial-using-pytorch-lightning/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-using-pytorch-lightning)|[1.train-single-node.ipynb](tutorials/using-pytorch-lightning/1.train-single-node.ipynb)<br>[2.log-with-tensorboard.ipynb](tutorials/using-pytorch-lightning/2.log-with-tensorboard.ipynb)<br>[3.log-with-mlflow.ipynb](tutorials/using-pytorch-lightning/3.log-with-mlflow.ipynb)<br>[4.train-multi-node-ddp.ipynb](tutorials/using-pytorch-lightning/4.train-multi-node-ddp.ipynb)|learn how to train and log metrics with [PyTorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning)
[using-rapids](tutorials/using-rapids)|[![using-rapids](https://github.com/Azure/azureml-examples/workflows/tutorial-using-rapids/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-using-rapids)|[1.train-and-hpo.ipynb](tutorials/using-rapids/1.train-and-hpo.ipynb)<br>[2.train-multi-gpu.ipynb](tutorials/using-rapids/2.train-multi-gpu.ipynb)|learn how to accelerate PyData tools (numpy, pandas, scikit-learn, etc) on NVIDIA GPUs with [rapids](https://github.com/rapidsai)
[using-xgboost](tutorials/using-xgboost)|[![using-xgboost](https://github.com/Azure/azureml-examples/workflows/tutorial-using-xgboost/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-using-xgboost)|[1.local-eda.ipynb](tutorials/using-xgboost/1.local-eda.ipynb)<br>[2.distributed-cpu.ipynb](tutorials/using-xgboost/2.distributed-cpu.ipynb)|learn how to use [XGBoost](https://github.com/dmlc/xgboost) on Azure
**Notebooks** ([notebooks](notebooks))
path|status|description
-|-|-
[notebooks/train-lightgbm-local.ipynb](notebooks/train-lightgbm-local.ipynb)|[![train-lightgbm-local](https://github.com/Azure/azureml-examples/workflows/notebook-tll/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Anotebook-tll)|use mlflow for tracking local notebook experimentation in the cloud
[notebooks/train-lightgbm-local.ipynb](notebooks/train-lightgbm-local.ipynb)|[![train-lightgbm-local](https://github.com/Azure/azureml-examples/workflows/notebook-train-lightgbm-local/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Anotebook-train-lightgbm-local)|use mlflow for tracking local notebook experimentation in the cloud
**Train** ([workflows/train](workflows/train))
@ -111,5 +111,5 @@ This project has adopted the [Microsoft Open Source Code of Conduct](https://ope
## Reference
- [GitHub Template](https://github.com/Azure/azureml-template)
- [Cheat Sheet, VSCode Snippets, and Templates](https://azure.github.io/azureml-web)
- [Open Source Azure ML Cheat Sheet](https://azure.github.io/azureml-web)
- [Azure Machine Learning Documentation](https://docs.microsoft.com/azure/machine-learning)

Просмотреть файл

@ -48,7 +48,7 @@ A lightweight template repository for automating the ML lifecycle can be found [
|`.cloud`|cloud templates|
|`.github`|GitHub specific files like Actions workflow yaml definitions and issue templates|
|`notebooks`|interactive jupyter notebooks for iterative ML development|
|`tutorials`|self-contained directories of end-to-end tutorials|
|`tutorials`|self-contained directories of tutorials|
|`workflows`|self-contained directories of job to be run, organized by scenario then tool then project|
## Examples

Просмотреть файл

@ -67,12 +67,11 @@ def write_readme(tutorials, notebooks, workflows):
nbs = [f"[{nb}]({tutorial}/{nb})" for nb in nbs]
nbs = "<br>".join(nbs)
# get tutorial name and initials
# get tutorial name
name = tutorial.split("/")[-1]
initials = "".join(word[0][0] for word in name.split("-"))
# build entries for tutorial table
status = f"[![{name}](https://github.com/Azure/azureml-examples/workflows/tutorial-{initials}/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-{initials})"
status = f"[![{name}](https://github.com/Azure/azureml-examples/workflows/tutorial-{name}/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Atutorial-{name})"
description = "*no description*"
try:
with open(f"{tutorial}/README.md", "r") as f:
@ -89,16 +88,15 @@ def write_readme(tutorials, notebooks, workflows):
# process notebooks
for notebook in notebooks:
# get notebook name and initials
# get notebook name
name = notebook.split("/")[-1].replace(".ipynb", "")
initials = "".join(word[0][0] for word in name.split("-"))
# read in notebook
with open(notebook, "r") as f:
data = json.load(f)
# build entries for notebook table
status = f"[![{name}](https://github.com/Azure/azureml-examples/workflows/notebook-{initials}/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Anotebook-{initials})"
status = f"[![{name}](https://github.com/Azure/azureml-examples/workflows/notebook-{name}/badge.svg)](https://github.com/Azure/azureml-examples/actions?query=workflow%3Anotebook-{name})"
description = "*no description*"
try:
if "description: " in str(data["cells"][0]["source"]):
@ -161,12 +159,11 @@ def write_readme(tutorials, notebooks, workflows):
def write_workflows(notebooks, workflows):
# process notebooks
for notebook in notebooks:
# get notebook name and initials
# get notebook name
name = notebook.split("/")[-1].replace(".ipynb", "")
initials = "".join(word[0][0] for word in name.split("-"))
# write workflow file
write_notebook_workflow(notebook, name, initials)
write_notebook_workflow(notebook, name)
# process workflows
for workflow in workflows:
@ -215,9 +212,9 @@ def format_code():
os.system("black-nb --clear-output .")
def write_notebook_workflow(notebook, name, initials):
def write_notebook_workflow(notebook, name):
creds = "${{secrets.AZ_AE_CREDS}}"
workflow_yaml = f"""name: notebook-{initials}
workflow_yaml = f"""name: notebook-{name}
on:
schedule:
- cron: "0 0/2 * * *"
@ -226,7 +223,7 @@ on:
- main
paths:
- {notebook}
- .github/workflows/notebook-{initials}.yml
- .github/workflows/notebook-{name}.yml
jobs:
build:
runs-on: ubuntu-latest
@ -251,7 +248,7 @@ jobs:
run: papermill {notebook} out.ipynb -k python\n"""
# write workflow
with open(f".github/workflows/notebook-{initials}.yml", "w") as f:
with open(f".github/workflows/notebook-{name}.yml", "w") as f:
f.write(workflow_yaml)

Просмотреть файл

@ -10,5 +10,5 @@ This project has adopted the [Microsoft Open Source Code of Conduct](https://ope
## Reference
- [GitHub Template](https://github.com/Azure/azureml-template)
- [Cheat Sheet, VSCode Snippets, and Templates](https://azure.github.io/azureml-web)
- [Open Source Azure ML Cheat Sheet](https://azure.github.io/azureml-web)
- [Azure Machine Learning Documentation](https://docs.microsoft.com/azure/machine-learning)