added checklists, tidy up, fixes broken links, misc
fix reference links and renamed submodule renamed submodule revert renaming submodule added submodule back
This commit is contained in:
changhow
Родитель
a190f2aab8
Коммит
5ea7950cc6
|
|
@ -1,6 +1,7 @@
|
|||
[submodule "3-Deploy/azure-machine-learning-terraform"]
|
||||
path = 3-Deploy/azure-machine-learning-terraform
|
||||
url = https://github.com/csiebler/azure-machine-learning-terraform
|
||||
|
||||
[submodule "4-Migrate/dstoolkit-mlops-base"]
|
||||
path = 4-Migrate/dstoolkit-mlops-base
|
||||
url = https://github.com/microsoft/dstoolkit-mlops-base
|
||||
|
|
|
|||
|
|
@ -3,4 +3,6 @@ sort: 3
|
|||
---
|
||||
# Skilling Plan
|
||||
|
||||
Customized skilling plan for roles
|
||||
Customized skilling plan for roles
|
||||
|
||||
> **NEXT:** Review the [checklist](/1-MLOpsFoundation/checklist.md) to see if your team are ready to move on to [design and provisioning AML infrasuctures](/2-Design/README.md) for your Machine Learning services.
|
||||
|
|
@ -4,12 +4,15 @@ sort: 1
|
|||
# ML Ops Foundations
|
||||
|
||||
This section focuses on Pre-kick off enablement on core Dev Ops, ML Ops fundamentals , the why (key motivations) and considerations relevant to all project roles. Topics such as:
|
||||
* Dev Ops Overview:
|
||||
* Key principles
|
||||
* Culture
|
||||
* CI/CD
|
||||
* What is Infrastructure as Code
|
||||
* ML Ops Overview:
|
||||
* ML Ops v. Dev Ops
|
||||
* Key Principles and Maturity Model
|
||||
* Skills Roles and Responsibilities
|
||||
|
||||
1. [**DevOps Overview**](0-DevOpsOverview/README.md). Understanding the key principles of Dev Ops, development cultures, CI/CD concepts, and the adaptation of Infrastructure as code (IaC).
|
||||
|
||||
2. [**MLOps Overview**](1-MLOpsOverview/README.md). Understand the difference between ML Ops and Dev Ops, the ML Ops maturity models of your organization, key principles of adopting ML Ops.
|
||||
|
||||
3. [**Skills Roles and Responsibilities of your team**](2-SkillsRolesAndResponsibilities/README.md). Identify the distinct roles and teams required for the ML Ops processes and understand different roles work together to manage the Machine Learning project.
|
||||
|
||||
## Deliverables
|
||||
* [Review the checklist](checklist.md)
|
||||
|
||||
> **GET STARTED:**
|
||||
> Start by understanding the key principles of Dev Ops, development cultures, CI/CD concepts, and the adaptation of Infrastructure as code (IaC) [here](0-DevOpsOverview/README.md).
|
||||
|
|
|
|||
|
|
@ -1,7 +1,15 @@
|
|||
# Checklist
|
||||
# MLOps Foundation Checklist
|
||||
|
||||
Before designing and provisioning Azure Infrastructure for your Machine Learning services:
|
||||
|
||||
- [ ] [Do you understand DevOps? Does your organisation have a DevOps Capability?](./0-DevOpsOverview/README.md)
|
||||
|
||||
- [ ] [Do you understand MLOps? Do you understand the difference between MLOps and DevOps?](./1-MLOpsOverview/README.md)
|
||||
|
||||
- [ ] [Does your Organisation have a clear Roles and Responsiblities for your team that align with the Roles and responsibilities that are required to support the growth journey of machine learning?](./2-SkillsRolesAndResponsibilities/README.md/#skills-roles--responsibilities)
|
||||
|
||||
- [ ] [Do you know where your current team and orgnaisation capability and position on the ML Maturity Model?](./1-MLOpsOverview/2-MLOpsMaturityModel.md#maturity-model)
|
||||
- [ ] [Is your team enabled and have the right culture and delivery model to deploy machine learning initiatives and projects?](./2-SkillsRolesAndResponsibilities/1-AdoptingDSProcess.md)
|
||||
|
||||
- [ ] [Is your team enabled and have the right culture and delivery model to deploy machine learning initiatives and projects?](./2-SkillsRolesAndResponsibilities/1-AdoptingDSProcess.md)
|
||||
|
||||
> **NEXT:** If your team are ready, move on to [design and provisioning AML infrasuctures](../2-Design/README.md) for your Machine Learning services.
|
||||
|
|
@ -28,47 +28,48 @@ The components and procedures dicussed in this section are as follow:
|
|||
<tr>
|
||||
<td>
|
||||
<ol>
|
||||
<li> <a href="#1-workspace"> Workspace</a>
|
||||
<li> <a href="#2-managed-resources"> Managed Resources</a>
|
||||
<ul style="list-style-type: upper-alpha;"><li><a href="#computes"> Computes</a></ul>
|
||||
<li> <a href="#3-assets"> Assets</a>
|
||||
<li> <a href="#1-workspace"> Workspace</a></li>
|
||||
<li> <a href="#2-managed-resources"> Managed Resources</a></li>
|
||||
<ul style="list-style-type: upper-alpha;">
|
||||
<li> <a href="#datasets-and-datastores"> Datasets and Datastores </a>
|
||||
<li> <a href="#environments"> Environments </a>
|
||||
<li> <a href="#experiments"> Experiments </a>
|
||||
<li><a href="#computes"> Computes</a></li>
|
||||
</ul>
|
||||
<li> <a href="#3-assets"> Assets</a></li>
|
||||
<ul style="list-style-type: upper-alpha;">
|
||||
<li> <a href="#datasets-and-datastores"> Datasets and Datastores </a></li>
|
||||
<li> <a href="#environments"> Environments </a></li>
|
||||
<li> <a href="#experiments"> Experiments </a></li>
|
||||
<ul style="list-style-type: lower-roman;">
|
||||
<li> <a href="#runs"> Runs </a>
|
||||
<li> <a href="#training-workflow"> Training Workflow </a>
|
||||
<li> <a href="#runs"> Runs </a></li>
|
||||
<li> <a href="#training-workflow"> Training Workflow </a></li>
|
||||
</ul>
|
||||
<li> <a href="#models"> Models </a>
|
||||
<li> <a href="#models"> Models </a></li>
|
||||
<ul style="list-style-type: lower-roman;">
|
||||
<li> <a href="#model-registry"> Model Registry </a>
|
||||
<li> <a href="#model-registry"> Model Registry </a></li>
|
||||
</ul>
|
||||
</ul>
|
||||
</ol>
|
||||
</td>
|
||||
<!-- Procedures -->
|
||||
<td>
|
||||
<ol>
|
||||
<li> <a href="#1-deployment"> Deployment </a>
|
||||
<li> <a href="#1-deployment"> Deployment </a></li>
|
||||
<ul style="list-style-type: upper-alpha;">
|
||||
<li><a href="#endpoints"> Endpoints </a>
|
||||
<li><a href="#endpoints"> Endpoints </a></li>
|
||||
<ul style="list-style-type: lower-roman;">
|
||||
<li> <a href="#web-service-endpoints"> Web Service Endpoints </a>
|
||||
<li> <a href="#real-time-endpoints"> Real-time Endpoints </a>
|
||||
<li> <a href="#pipeline-endpoints"> Pipeline Endpoints </a>
|
||||
<li> <a href="#web-service-endpoints"> Web Service Endpoints </a></li>
|
||||
<li> <a href="#real-time-endpoints"> Real-time Endpoints </a></li>
|
||||
<li> <a href="#pipeline-endpoints"> Pipeline Endpoints </a></li>
|
||||
</ul>
|
||||
</ul>
|
||||
<li> <a href="#2-automation"> Automation </a>
|
||||
<li> <a href="#2-automation"> Automation </a></li>
|
||||
<ul style="list-style-type: upper-alpha;">
|
||||
<li> <a href="#azure-machine-learning-cli"> Azure Machine Learning CLI </a>
|
||||
<li> <a href="#ml-pipelines"> ML Pipelines </a>
|
||||
<li> <a href="#azure-machine-learning-cli"> Azure Machine Learning CLI </a></li>
|
||||
<li> <a href="#ml-pipelines"> ML Pipelines </a></li>
|
||||
</ul>
|
||||
<li> <a href="#3-monitoring-and-logging"> Monitoring and Logging </a>
|
||||
<li> <a href="#4-interacting-with-workspace"> Interacting with Workspace </a>
|
||||
<li> <a href="#3-monitoring-and-logging"> Monitoring and Logging </a></li>
|
||||
<li> <a href="#4-interacting-with-workspace"> Interacting with Workspace </a></li>
|
||||
<ul style="list-style-type: upper-alpha;">
|
||||
<li> <a href="#azure-machine-learning-studio"> Azure Machine Learning Studio </a>
|
||||
<li> <a href="#programming-tools"> Programming Tools </a>
|
||||
<li> <a href="#azure-machine-learning-studio"> Azure Machine Learning Studio </a></li>
|
||||
<li> <a href="#programming-tools"> Programming Tools </a></li>
|
||||
</ul>
|
||||
</ol>
|
||||
</td>
|
||||
|
|
|
|||
|
|
@ -172,6 +172,8 @@ For more information on using Azure Pipelines with Azure Machine Learning, see t
|
|||
|
||||
* [Continuous integration and deployment of ML models with Azure Pipelines](https://docs.microsoft.com/en-us/azure/devops/pipelines/targets/azure-machine-learning)
|
||||
* [Azure Machine Learning MLOps](https://aka.ms/mlops) repository.
|
||||
* [Azure Machine Learning MLOpsPython](https://github.com/Microsoft/MLOpspython) repository.
|
||||
* [Azure Machine Learning MLOps Python](https://github.com/Microsoft/MLOpspython) repository.
|
||||
|
||||
You can also use Azure Data Factory to create a data ingestion pipeline that prepares data for use with training. For more information, see [Data ingestion pipeline](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-cicd-data-ingestion).
|
||||
You can also use Azure Data Factory to create a data ingestion pipeline that prepares data for use with training. For more information, see [Data ingestion pipeline](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-cicd-data-ingestion).
|
||||
|
||||
>**NEXT:** Understand how to manage your Azure infrastructures services, see [Infrastructure Service Management](3-InfrastructureServiceManagement/README.md).
|
||||
|
|
@ -39,16 +39,16 @@ Listed below are the service options that you should consider for each stage of
|
|||
|
||||
### Service options
|
||||
|
||||
| Stage | Service options |
|
||||
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Experimentation | [Azure Machine Learning Notebook VMs](https://azure.microsoft.com/blog/three-things-to-know-about-azure-machine-learning-notebook-vm/)<br>[Databricks Notebooks](https://docs.databricks.com/notebooks/index.html)<br>[Azure Machine Learning Experiment for Python SDK](/python/api/overview/azure/ml/)<br>[Azure Data Science Virtual Machine (DSVM)](/azure/machine-learning/data-science-virtual-machine/) |
|
||||
| Overall Orchestration / Scheduling | [Azure Logic Apps](/azure/logic-apps/logic-apps-overview)<br>[Azure Data Factory](/azure/data-factory/introduction)<br>[Azure Machine Learning Pipelines](/azure/machine-learning/concept-ml-pipelines)<br>[Azure DevOps](/azure/devops/)<br> |
|
||||
| Data Transfer | [Azure Data Factory Copy Activity](/azure/data-factory/copy-activity-overview)<br>[Azure Machine Learning DataTransferStep](/python/api/azureml-pipeline-steps/azureml.pipeline.steps.data_transfer_step.datatransferstep) |
|
||||
| Compute | [Databricks](/azure/azure-databricks/what-is-azure-databricks)<br>[Azure Machine Learning Compute](/azure/machine-learning/concept-compute-instance) |
|
||||
| Tracking / Versioning options | Experiment/Hyper-tuning Tracking:<br><ul>[Azure Machine Learning Experiments](/azure/machine-learning/studio/create-experiment)<br>[Databricks and MLFLow Tracking](https://docs.databricks.com/applications/mlflow/quick-start.html)</ul><br>[Data Versioning/Data Drift: Azure Machine Learning Datasets](/azure/machine-learning/how-to-version-track-datasets)<br><br>Model Versioning:<ul>[Azure Machine Learning Model Management Service](/azure/machine-learning/concept-model-management-and-deployment)<br>[Databricks and MLFlow Model Registry](https://databricks.com/blog/2019/10/17/introducing-the-mlflow-model-registry.html)<br></ul> |
|
||||
| Model Training | [Azure Machine Learning Pipelines](/azure/machine-learning/concept-ml-pipelines)<br>[Databricks](https://docs.databricks.com/data/index.html) |
|
||||
| Model Deployment | [Batch Scoring in Azure Machine Learning Pipeline](/azure/machine-learning/tutorial-pipeline-batch-scoring-classification)<br>[Real-time Deployment in Azure Machine Learning Service](/azure/machine-learning/how-to-deploy-and-where)<ul>[Azure Kubernetes Service (AKS)](/azure/aks/intro-kubernetes)<br>[Azure Container Instance](/azure/container-instances/)<br>[Azure App Service](/azure/app-service/)<br>[Azure Functions](/azure/azure-functions/)<br>[IoT Edge](/azure/iot-edge/about-iot-edge)<br>[Azure Machine Learning Model Deployment](/azure/machine-learning/how-to-deploy-and-where)</ul> |
|
||||
| Monitoring | [Azure Monitor](/azure/azure-monitor/overview)<ul>[Application Insights](/azure/azure-monitor/app/app-insights-overview)<br>[Azure Dashboards](/azure/azure-monitor/learn/tutorial-app-dashboards)</ul>[Power BI](/power-bi/service-azure-and-power-bi) |
|
||||
| Stage | Service options |
|
||||
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| Experimentation | [Azure Machine Learning Notebook VMs](https://azure.microsoft.com/blog/three-things-to-know-about-azure-machine-learning-notebook-vm/)<br>[Databricks Notebooks](https://docs.databricks.com/notebooks/index.html)<br>[Azure Machine Learning Experiment for Python SDK](https://docs.microsoft.com/en-us/python/api/overview/azure/ml/?view=azure-ml-py)<br>[Azure Data Science Virtual Machine (DSVM)](https://docs.microsoft.com/azure/machine-learning/data-science-virtual-machine/) |
|
||||
| Overall Orchestration / Scheduling | [Azure Logic Apps](https://docs.microsoft.com/azure/logic-apps/logic-apps-overview)<br>[Azure Data Factory](https://docs.microsoft.com/azure/data-factory/introduction)<br>[Azure Machine Learning Pipelines](https://docs.microsoft.com/azure/machine-learning/concept-ml-pipelines)<br>[Azure DevOps](https://docs.microsoft.com/azure/devops/)<br> |
|
||||
| Data Transfer | [Azure Data Factory Copy Activity](https://docs.microsoft.com/azure/data-factory/copy-activity-overview)<br>[Azure Machine Learning DataTransferStep](https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.data_transfer_step.datatransferstep) |
|
||||
| Compute | [Databricks](https://docs.microsoft.com/azure/azure-databricks/what-is-azure-databricks)<br>[Azure Machine Learning Compute](https://docs.microsoft.com/azure/machine-learning/concept-compute-instance) |
|
||||
| Tracking / Versioning options | Experiment/Hyper-tuning Tracking:<br><ul>[Azure Machine Learning Experiments](https://docs.microsoft.com/azure/machine-learning/studio/create-experiment)<br>[Databricks and MLFLow Tracking](https://docs.databricks.com/applications/mlflow/quick-start.html)</ul><br>[Data Versioning/Data Drift: Azure Machine Learning Datasets](https://docs.microsoft.com/azure/machine-learning/how-to-version-track-datasets)<br><br>Model Versioning:<ul>[Azure Machine Learning Model Management Service](https://docs.microsoft.com/azure/machine-learning/concept-model-management-and-deployment)<br>[Databricks and MLFlow Model Registry](https://databricks.com/blog/2019/10/17/introducing-the-mlflow-model-registry.html)<br></ul> |
|
||||
| Model Training | [Azure Machine Learning Pipelines](https://docs.microsoft.com/azure/machine-learning/concept-ml-pipelines)<br>[Databricks](https://docs.databricks.com/data/index.html) |
|
||||
| Model Deployment | [Batch Scoring in Azure Machine Learning Pipeline](https://docs.microsoft.com/azure/machine-learning/tutorial-pipeline-batch-scoring-classification)<br>[Real-time Deployment in Azure Machine Learning Service](https://docs.microsoft.com/azure/machine-learning/how-to-deploy-and-where)<ul>[Azure Kubernetes Service (AKS)](https://docs.microsoft.com/azure/aks/intro-kubernetes)<br>[Azure Container Instance](https://docs.microsoft.com/azure/container-instances/)<br>[Azure App Service](https://docs.microsoft.com/azure/app-service/)<br>[Azure Functions](https://docs.microsoft.com/azure/azure-functions/)<br>[IoT Edge](https://docs.microsoft.com/azure/iot-edge/about-iot-edge)<br>[Azure Machine Learning Model Deployment](https://docs.microsoft.com/azure/machine-learning/how-to-deploy-and-where)</ul> |
|
||||
| Monitoring | [Azure Monitor](https://docs.microsoft.com/azure/azure-monitor/overview)<ul>[Application Insights](https://docs.microsoft.com/azure/azure-monitor/app/app-insights-overview)<br>[Azure Dashboards](https://docs.microsoft.com/azure/azure-monitor/learn/tutorial-app-dashboards)</ul>[Power BI](https://docs.microsoft.com/en-us/power-bi/connect-data/power-bi/service-azure-and-power-bi) |
|
||||
|
||||
## Use No Code or Code implementation approach
|
||||
|
||||
|
|
@ -70,13 +70,13 @@ If you don't want to code your own solutions, a set of tools is available for bu
|
|||
|
||||
The primary issue you’ll come across here is that you must work within the constraints of the services. However, if your use case fits within these limitations, these services could be a good solution for you. They're always evolving and their capabilities will expand over time. So you should familiarize yourself with their latest features at the time you consider them. This diagram summarizes the process for the No Code option.
|
||||
|
||||
|
||||
|
||||
|
||||
### Code
|
||||
|
||||
If you want to code or need the flexibility that a coded solution offers, all of the options described have a “code-like” interface. The options also have a representation of processing logic that you can export to JSON or YAML format and check in the exported files to a code repository. From there, deployment is handled through Azure DevOps or scripts. This diagram summarizes the Code option process.
|
||||
|
||||
|
||||
|
||||
|
||||
## Experimentation: Notebooks vs. Python/R scripts
|
||||
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ Here's the authentication process for Azure Machine Learning using multi-factor
|
|||
1. The client presents the token to Azure Resource Manager and to all Azure Machine Learning.
|
||||
1. Azure Machine Learning provides a Machine Learning service token to the user compute target (for example, Azure Machine Learning compute cluster). This token is used by the user compute target to call back into the Machine Learning service after the run is complete. The scope is limited to the workspace.
|
||||
|
||||
[](media/authentication.png#lightbox)
|
||||
[](media/authentication.png#lightbox)
|
||||
|
||||
Each workspace has an associated system-assigned [managed identity](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview) that has the same name as the workspace. This managed identity is used to securely access resources used by the workspace. It has the following Azure RBAC permissions on associated resources:
|
||||
|
||||
|
|
|
|||
|
|
@ -29,10 +29,10 @@ Hardware requirements for your training workload might vary from project to proj
|
|||
|
||||
You might not know yet what your compute requirements are. In this scenario, we recommend starting with either of the following cost effective default options. These options are for lightweight testing and for training workloads.
|
||||
|
||||
| **Type** | **Virtual machine size** | **Specs** |
|
||||
| --- | --- | --- |
|
||||
| CPU | Standard\_DS3\_v2 | 4 cores, 14 gigabytes (GB) RAM, 28-GB storage |
|
||||
| GPU | Standard\_NC6 | 6 cores, 56 gigabytes (GB) RAM, 380-GB storage, NVIDIA Tesla K80 GPU |
|
||||
| **Type** | **Virtual machine size** | **Specs** |
|
||||
| -------- | ------------------------ | -------------------------------------------------------------------- |
|
||||
| CPU | Standard\_DS3\_v2 | 4 cores, 14 gigabytes (GB) RAM, 28-GB storage |
|
||||
| GPU | Standard\_NC6 | 6 cores, 56 gigabytes (GB) RAM, 380-GB storage, NVIDIA Tesla K80 GPU |
|
||||
|
||||
To get the best VM size for your scenario, it might consist of trial and error. Here are several aspects to consider.
|
||||
|
||||
|
|
@ -97,7 +97,7 @@ Based on the insights from the monitoring details, you can better plan or adjust
|
|||
|
||||
You can access these metrics directly from the Azure portal. Go to your Azure Machine Learning workspace, and select *Metrics* under the monitoring section on the left panel. Then, you can select details on what you would like to view, such as metrics, aggregation, and time period. For more information, see [Monitor Azure Machine Learning](https://docs.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning) documentation page.
|
||||
|
||||
|
||||
|
||||
|
||||
### Switch between local, single-node, and multi-node cloud compute while you develop
|
||||
|
||||
|
|
@ -216,4 +216,6 @@ When you [pick a region for your compute](https://azure.microsoft.com/global-inf
|
|||
|
||||
## Learn more
|
||||
|
||||
[Track costs across business units, environments, or projects by using the Cloud Adoption Framework](https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/track-costs)
|
||||
[Track costs across business units, environments, or projects by using the Cloud Adoption Framework](https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/track-costs)
|
||||
|
||||
> **NEXT:** Review the [checklist](../checklist.md) to see if your team are ready to move on to [deploying your AML infrastructures](../../3-Deploy/README.md) for your Machine Learning services.
|
||||
|
|
@ -1,19 +1,29 @@
|
|||
---
|
||||
sort: 2
|
||||
---
|
||||
# Design and Provision Infrastructure
|
||||
# Design and Provision AML Infrastructure
|
||||
Azure Machine Learning is a cloud service for accelerating and managing the machine learning project lifecycle. Machine learning professionals, data scientists, and engineers can use it in their day-to-day workflows: Train and deploy models, and manage MLOps.
|
||||
|
||||
You can create a model in Azure Machine Learning or use a model built from an open-source platform, such as Pytorch, TensorFlow, or scikit-learn. MLOps tools help you monitor, retrain, and redeploy models.
|
||||
Before deployment with Azure resources, development teams should:
|
||||
1. [Understand the architecture and design concepts of Azure Machine Learning](1-MLOpsArchitectureConcepts.md).
|
||||
|
||||
2. [Learn about model management, deployment, lineage and monitoring with Azure Machine Learning](2-ModelManagement.md).
|
||||
3. [Understand the technology choices available](3-AMLTechChoices.md).
|
||||
4. [Understand the technology Selection criteria for edge deployment](4-EdgeDeployment.md).
|
||||
5. [Understand the Enterprise security and governance for Azure Machine Learning](5-EnterpriseSecurity&Governance.md)
|
||||
6. [Set up authentication for Azure Machine Learning resources and workflows](6-Authentication.md)
|
||||
7. [Manage access to an Azure Machine Learning workspace](7-how-to-assign-roles.md)
|
||||
|
||||
3. [Understand the technology choices available](3-InfrastructureServiceManagement/1-AMLTechChoices.md).
|
||||
|
||||
4. [Understand the technology Selection criteria for edge deployment](3-InfrastructureServiceManagement/2-EdgeDeployment.md).
|
||||
|
||||
5. [Understand the Enterprise security and governance for Azure Machine Learning](3-InfrastructureServiceManagement/3-EnterpriseSecurity%26Governance.md)
|
||||
|
||||
6. [Set up authentication for Azure Machine Learning resources and workflows](3-InfrastructureServiceManagement/4-Authentication.md)
|
||||
|
||||
7. [Manage access to an Azure Machine Learning workspace](3-InfrastructureServiceManagement/5-how-to-assign-roles.md)
|
||||
|
||||
8. [Understand cost management of the Azure Machine Learning services](3-InfrastructureServiceManagement/6-cost-management.md)
|
||||
|
||||
## Deliverables
|
||||
* Review the checklist
|
||||
* [Review the checklist](checklist.md)
|
||||
|
||||
> **GET STARTED:**
|
||||
> Start by understanding the architecture and design concepts of Azure Machine Learning [here](1-MLOpsArchitectureConcepts.md).
|
||||
|
|
@ -1,3 +1,21 @@
|
|||
---
|
||||
sort: 4
|
||||
---
|
||||
# Design and Provision AML Infrastructure Checklist
|
||||
|
||||
Before deploying your Azure Infrastructure:
|
||||
|
||||
- [ ] [Does your team have an understanding of the AML Architecture and design concepts?](1-MLOpsArchitectureConcepts.md)
|
||||
|
||||
- [ ] [Does your team have an understanding of MLOps with Azure AML?](2-ModelManagement.md)
|
||||
|
||||
- [ ] [Have you decided on the technology choices that are fit for your project?](3-InfrastructureServiceManagement/1-AMLTechChoices.md)
|
||||
|
||||
- [ ] Do you have an agreed and approved Solution Design?
|
||||
|
||||
- [ ] [Does your Solution Design have the right security controls in place (vNET Integration, Use of Private endpoint, etc?](3-InfrastructureServiceManagement/3-EnterpriseSecurity%26Governance.md)
|
||||
|
||||
- [ ] [Does the solution design specify the rules for Access Control using RBAC?](3-InfrastructureServiceManagement/5-how-to-assign-roles.md)
|
||||
|
||||
- [ ] [Does your Solution design adhere to Microsoft CAF for MLOps and to MLOps best practices?](https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/ai-machine-learning-mlops#machine-learning-devops-mlops-best-practices-with-azure-machine-learning)
|
||||
|
||||
- [ ] [Have you configured Cost Control Measures?](3-InfrastructureServiceManagement/6-cost-management.md)
|
||||
|
||||
> **NEXT:** If your team are ready, move on to [deploying your AML infrastructures](../3-Deploy/README.md) for your Machine Learning services.
|
||||
|
|
@ -65,3 +65,7 @@ Alternatively, you can also deploy your development environments with [YAML Pipe
|
|||
10. Select and view the existing YAML in your default editor and make changes.
|
||||
11. Commit the YAML file to the main branch.
|
||||
12. Azure DevOps will automatically start a pipeline run. Wait for the run to finish.
|
||||
|
||||
> **NEXT:** Review the [checklist](checklist.md) to see if your team is ready for continuous integration and development (CI/CD) of your Machine Learning services development.
|
||||
>
|
||||
> **ADDITIONAL INFO:** Your team can also accelerate the migration of your existing model to AML by referring to the materials and templates [here](../4-Migrate/README.md).
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
sort: 3
|
||||
---
|
||||
# Deployment of Azure Machine Learning Infrastructure
|
||||
# AML Infrastructure Deployment
|
||||
|
||||
The deployment of Azure Machine Learning Infrastructures requires several steps:
|
||||
1. [**Set up local development environment**](1-SetupLocalEnvironment.md).
|
||||
|
|
@ -26,5 +26,6 @@ Alternatively, to roll out a complete Azure Machine Learning enterprise environm
|
|||
[Azure Machine Learning Enterprise Terraform Example](azure-machine-learning-terraform/README.md).
|
||||
|
||||
## Deliverables
|
||||
* Full deployed services on Azure using Automated pipelines
|
||||
* Review the checklist
|
||||
* [Review the checklist](checklist.md)
|
||||
|
||||
> **ADDITIONAL INFO:** Your team can accelerate the migration of your existing model to AML by referring to the materials and templates [here](../4-Migrate/README.md).
|
||||
|
|
@ -1,3 +1,16 @@
|
|||
---
|
||||
sort: 4
|
||||
---
|
||||
# AML Infrastructure Deployment Checklist
|
||||
For continuous integration and development (CI/CD) of your Machine Learning services development:
|
||||
|
||||
- [ ] [Have you setup your local environment to connect to your AML workspaces?](1-SetupLocalEnvironment.md)
|
||||
|
||||
- [ ] [Have you organised workspaces for the different teams/projects in Azure ML?](2-OrganizeAMLEnvironment.md)
|
||||
|
||||
- [ ] [Have you designed and deployed separate environments for Dev, Test, PROD](3-CreateSeparateEnvironments.md#creating-separate-environments-for-development)
|
||||
|
||||
- [ ] [Are you able to deploy chances using the automated deployment pipelines?](3-CreateSeparateEnvironments.md#using-azure-pipeline-for-separate-development-environment-deployment)
|
||||
|
||||
- [ ] [Have you deployed Azure ML service and related services on Azure?](README.md#quickstart)
|
||||
|
||||
- [ ] [Have you automated the deployment of these services using Infrastructure-as-code (IaC)](https://github.com/vNEXTAU/azureml-ops-accelerator/tree/dev/3-Deploy/ARMTEMPLATES)
|
||||
|
||||
> **ADDITIONAL INFO:** Your team can accelerate the migration of your existing model to AML by referring to the materials and templates [here](../4-Migrate/README.md).
|
||||
|
|
@ -3,6 +3,16 @@ sort: 4
|
|||
---
|
||||
# Migrate and Operate
|
||||
|
||||
This section provides a reference implementation of MLOps using AML that can be used to move your existing models and includes:
|
||||
* Code artefacts to move your first workload (existing model) onto AML and "MLOPs" it across the newly created environments
|
||||
* Operational best practices
|
||||
This section helps accelerate the migration of your existing models to Azure ML services.
|
||||
|
||||
Reference materials for Azure ML key concepts, best practices implementation of MLOps, and the template to deploy Azure ML services are included:
|
||||
|
||||
1. [Azure ML Key Concepts](1-KeyAzureMLConceptsForOps.md). Understanding the key concepts of adopting Azure ML services before migration of your existing model to Azure ML services.
|
||||
|
||||
2. [Azure ML Best Practices](2-AMLBestPractices.md). Understanding the best practices when deploying ML Ops with Azure Machine Learning.
|
||||
|
||||
3. [Template for migration with guidance](dstoolkit-mlops-base/README.md). This repository provides guidance and templates to assist and accelerate the migration of your existing model to Azure ML services.
|
||||
|
||||
> **ADDITIONAL INFO:** To tune the ParallelRunStep (PRS) performance in Azure ML, follow [ParallelRunStep Performance Tuning Guide](/4-Migrate/3-PerformanceTunePRS.md).
|
||||
|
||||
|
||||
|
|
|
|||
|
До Ширина: | Высота: | Размер: 279 KiB После Ширина: | Высота: | Размер: 279 KiB |
46
README.md
46
README.md
|
|
@ -1,3 +1,6 @@
|
|||
---
|
||||
sort: 1
|
||||
---
|
||||
# Azure ML-Ops Project Accelerator
|
||||
|
||||
Guided accelerator consolidating best practice patterns, IaaC and AML code artefacts to provide reference IP to a support a baseline MLOps implementation on Azure leveraging Azure ML that can be delivered in approximately 12 weeks of project scope.
|
||||
|
|
@ -7,27 +10,28 @@ This repo is designed to be consumed 'documentation led', with the relevant IaaC
|
|||
Use the Table of Contents below to help you navigate to the section of repo that you are interested in based on your role or the stage of your project.
|
||||
|
||||
|
||||
| **Stage** | **Tasks** | **Roles** |
|
||||
| ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
|
||||
| [**1. MLOps Foundation**](1-MLOpsFoundation/README.md) | Understanding MLOps <ul><li> [What's DevOps?](1-MLOpsFoundation/0-DevOpsOverview/README.md) <li> [What's MLOps?](1-MLOpsFoundation/1-MLOpsOverview/README.md) </ul> | Everyone |
|
||||
| | Understand Maturity Model <ul> <li> [Determine Organization Capability Level](1-MLOpsFoundation/1-MLOpsOverview/2-MLOpsMaturityModel.md) <li> [Culture and Key Principles](1-MLOpsFoundation/2-SkillsRolesAndResponsibilities/1-AdoptingDSProcess.md) </ul> | Group Manager Team Lead Project Lead |
|
||||
| | Team Formation <ul> <li> [Skills, Roles, and Responsibilities](1-MLOpsFoundation/2-SkillsRolesAndResponsibilities/README.md)) <li> [Deciding on (agile) Delivery Model](1-MLOpsFoundation/2-SkillsRolesAndResponsibilities/1-AdoptingDSProcess.md) </ul> | Everyone |
|
||||
| | Deliverables <ul> <li> Review the Checklists </ul> | |
|
||||
| [**2. Design**](2-Design/README.md) | [Review AML Architecture and Design Concepts](2-Design/1-MLOpsArchitectureConcepts.md) | Team Lead Solution Architect |
|
||||
| | [Understanding MLOps with Azure AML](2-Design/2-ModelManagement.md) | Team Lead Solution Architect |
|
||||
| | [Make Technology Choices based on your use case and organisation's need](2-Design/3-AMLTechChoices.md) | Team Lead Solution Architect |
|
||||
| | Security Control for Service Infrastructure <ul> <li> [Use vNET Integrate & Private Link for AML](2-Design/5-EnterpriseSecurity%26Governance.md) </ul> | Solution Architect Azure Infrastructure Engineer Team Lead |
|
||||
| | Configuring Access Control <ul> <li> [Secure Access to AML with RBAC](2-Design/6-Authentication.md) <li> Use Custom Roles when required </ul> | Solution Architect Azure Infrastructure Engineer Team Lead |
|
||||
| | [Map Team Roles to RBAC](2-Design/7-how-to-assign-roles.md) | Team Lead Solution Architect |
|
||||
| | [Infrastructure Costs Management](2-Design/8-CostManagement.md) | Solution Architect Azure Infrastructure Engineer Team Lead Administrator |
|
||||
| | Deliverables <ul> <li> Approved Solution Design <li> Review the Checklists </ul> | |
|
||||
| [**3. Deploy** ](3-Deploy/README.md) | Accelerate Code Deployment for AML Services <ul> <li> Automate the Deployment of Resourcees <li> Update the Deployment Scripts to Match the approved Solution Design </ul> | Azure Infrastructure Engineer DevOps Engineer Team Lead |
|
||||
| | Setting up Local Environment for Development <ul> <li> [Install Tools](3-Deploy/1-SetupLocalEnvironment.md#installing-azure-cli) <li> [Connect to AML](3-Deploy/1-SetupLocalEnvironment.md#connect-to-aml) </ul> | Data Scientist MLOps Engineer Data Engineer |
|
||||
| | [Organise AML Environments](3-Deploy/2-OrganizeAMLEnvironment.md) | MLOps Engineer DevOps Engineer |
|
||||
| | [Creating Separate Environments (Dev, Test, Prod)](3-Deploy/3-CreateSeparateEnvironments.md) | MLOps Engineer DevOps Engineer |
|
||||
| | Deliverables <ul><li> Full Deployed Services on Azure using Automated Pipelines <li> Review the Checklists </ul> | |
|
||||
| [**4. Migrate**](4-Migrate/README.md) | Migrate existing Machine Learning Experiment to AML | MLOps Engineer |
|
||||
| | Deliverables <ul><li> Review the Checklists </ul> | |
|
||||
| **Stage** | **Tasks** | **Roles** |
|
||||
| ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
|
||||
| [**1. MLOps Foundation**](1-MLOpsFoundation/README.md) | Understanding MLOps <br><ul><li>[What's DevOps?](1-MLOpsFoundation/0-DevOpsOverview/README.md)</li><li>[What's MLOps?](1-MLOpsFoundation/1-MLOpsOverview/README.md)</li></ul> | Everyone |
|
||||
| | Understand Maturity Model <br><ul><li>[Determine Organization Capability Level](1-MLOpsFoundation/1-MLOpsOverview/2-MLOpsMaturityModel.md)</li><li>[Culture and Key Principles](1-MLOpsFoundation/1-MLOpsOverview/3-SevenMLOpsPrinciples.md)</li></ul> | Group Manager Team Lead Project Lead |
|
||||
| | Team Formation <br><ul><li>[Skills, Roles, and Responsibilities](1-MLOpsFoundation/2-SkillsRolesAndResponsibilities/README.md))</li><li>[Deciding on (agile) Delivery Model](1-MLOpsFoundation/2-SkillsRolesAndResponsibilities/1-AdoptingDSProcess.md)</li></ul> | Everyone |
|
||||
| | Deliverables <br><ul><li>[Review the Checklists](/1-MLOpsFoundation/checklist.md)</li></ul> | |
|
||||
| [**2. Design**](2-Design/README.md) | [Review AML Architecture and Design Concepts](2-Design/1-MLOpsArchitectureConcepts.md) | Team Lead Solution Architect |
|
||||
| | [Understanding MLOps with Azure AML](2-Design/2-ModelManagement.md) | Team Lead Solution Architect |
|
||||
| | [Make Technology Choices based on your use case and organisation's need](2-Design/3-AMLTechChoices.md) | Team Lead Solution Architect |
|
||||
| | Security Control for Service Infrastructure <br><ul><li>[Use vNET Integrate & Private Link for AML](2-Design/5-EnterpriseSecurity%26Governance.md)</li></ul> | Solution Architect Azure Infrastructure Engineer Team Lead |
|
||||
| | Configuring Access Control <br><ul><li>[Secure Access to AML with RBAC](2-Design/6-Authentication.md)</li></ul> | Solution Architect Azure Infrastructure Engineer Team Lead |
|
||||
| | [Map Team Roles to RBAC](2-Design/7-how-to-assign-roles.md)<ul><li>[Use Custom Roles when required](/2-Design/3-InfrastructureServiceManagement/5-how-to-assign-roles.md#create-custom-role)</li></ul> | Team Lead Solution Architect |
|
||||
| | [Infrastructure Costs Management](2-Design/8-CostManagement.md) | Solution Architect Azure Infrastructure Engineer Team Lead Administrator |
|
||||
| | Deliverables <br><ul><li>Approved Solution Design</li><li>[Review the Checklists](/2-Design/checklist.md)</li></ul> | |
|
||||
| [**3. Deploy** ](3-Deploy/README.md) | Accelerate Code Deployment for AML Services <br><ul><li>[Automate the Deployment of Resources](/3-Deploy/README.md#quickstart)</li><li>[Update the Deployment Scripts to Match the approved Solution Design](/3-Deploy/ARMTemplates/README.md)</li></ul> | Azure Infrastructure Engineer DevOps Engineer Team Lead |
|
||||
| | Setting up Local Environment for Development <br><ul><li>[Install Tools](3-Deploy/1-SetupLocalEnvironment.md#installing-azure-cli)</li><li>[Connect to AML](3-Deploy/1-SetupLocalEnvironment.md#connect-to-aml)</li></ul> | Data Scientist MLOps Engineer Data Engineer |
|
||||
| | [Organise AML Environments](3-Deploy/2-OrganizeAMLEnvironment.md) | MLOps Engineer DevOps Engineer |
|
||||
| | [Creating Separate Environments (Dev, Test, Prod)](3-Deploy/3-CreateSeparateEnvironments.md) | MLOps Engineer DevOps Engineer |
|
||||
| | Deliverables <br><ul><li>Full Deployed Services on Azure using Automated Pipelines</li><li>[Review the Checklists](/3-Deploy/checklist.md)</li></ul> | |
|
||||
| [**4. Migrate**](4-Migrate/README.md) | [Understanding AML Ops concepts](/4-Migrate/1-KeyAzureMLConceptsForOps.md) | MLOps Engineer |
|
||||
| | [Review AML Best Practices](/4-Migrate/2-AMLBestPractices.md) | MLOps Engineer |
|
||||
| | Deliverables <br><ul><li>[Review the Checklists](/4-Migrate/checklist.md)</li></ul> | |
|
||||
|
||||
# Contributing
|
||||
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче