diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..1633e49 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,11 @@ +# Contributing + +Thanks for considering contributing to this repository! + +While we are still in the early stages of development, we strongly suggest first coordinating with the developers before taking the time to write code and submit a pull request. Contributions in the form of issues and discussions are always welcome. + +## Pull Requests + +After coordinating through an issue or discussion, please submit a pull request with code changes. The team will review, test, and merge. + +We especially encourage code contributions for automating the development of the v2 solution accelerator. Until this is in place, the review and merge process will largely be manual. diff --git a/README.md b/README.md index e146e7c..d7ed798 100644 --- a/README.md +++ b/README.md @@ -1,66 +1,56 @@ -![Header](documentation/repositoryfiles/mlopsheader.jpg) - # Azure MLOps (v2) solution accelerator -MLOps or Machine Learning Ops is a set of practices that aims to automate and operationalise the deployment and maintenance of machine learning models across various stages of the lifecycle of a Data Science process. The purpose of an MLOps process is to drive efficiency, increase repeatability and predictability, enable reuse of code and drive consistency across projects. This enables Data Science teams to deploy Machine Learning models to production reliably and efficiently. +![Header](documentation/repositoryfiles/mlopsheader.jpg) -While MLOps has many overlapping concepts with DevOps and can be seen as a derivation of DevOps, it varies significantly from DevOps due to the nature of Data Science projects. The following Microsoft articles provide a perspective on MLOps from various viewpoints: +Welcome to the MLOps (v2) solution accelerator repository! This project is intended to serve as *the* starting point for MLOps implementation in Azure. -[MLOps using Azure Machine Learning](https://azure.microsoft.com/en-us/services/machine-learning/mlops/#features) +MLOps is a set of repeatable, automated, and collaborative workflows with best practices that empower teams of ML professionals to quickly and easily get their machine learning models deployed into production. You can learn more about MLOps here: -[Cloud Adoption Framework Guidance](https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/ai-machine-learning-mlops) +- [MLOps with Azure Machine Learning](https://azure.microsoft.com/services/machine-learning/mlops/#features) +- [Cloud Adoption Framework Guidance](https://docs.microsoft.com/azure/cloud-adoption-framework/ready/azure-best-practices/ai-machine-learning-mlops) +- [How: Machine Learning Operations](https://docs.microsoft.com/azure/machine-learning/concept-model-management-and-deployment) -[How: Machine Learning Operations](https://docs.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment) +## Prerequisites +1. An Azure subscription. If you don't have an Azure subscription, [create a free account](https://aka.ms/AzureMLFree) before you begin. -## Prerequisites: Repository Overview +## Project overview -The solution accelerator provides a modular end-to-end approach for MLOps in Azure based on pattern architectures. As each organization is unique, we do not expect that each pattern architecture will suit all organizations, however, Digital Natives or enterprises will be able to deploy an MLOps ecosystem fast, simple, reliable, modular, and secure. The time to product is measurable shorter under incerased scalability. +The solution accelerator provides a modular end-to-end approach for MLOps in Azure based on pattern architectures. As each organization is unique, solutions will often need to be customized to fit the organization's needs. -Following are the key principles that have been applied while building the accelerator: +The solution accelerator goals are: -     ✅**Simplicity** +- Simplicity +- Modularity +- Repeatability +- Collaboration +- Enterprise readiness + +It accomplishes these goals with a template-based approach for end-to-end data science, driving operational efficiency at each stage. You should be able to get up and running with the solution accelerator in a few hours. + +## 👤 Getting started: Azure Machine Learning - classical machine learning demo -     ✅**Segregation of duties & Security** +The demo follows the classical machine learning pattern with Azure Machine Learning. -     ✅**Reusability** - -     ✅**Modularity** - -     ✅**Enterprise readiness** - -This repo provides a templatised approach for the end-to-end Data Science process and focuses on driving efficiency at each stage. For example, it can take a significant amount of time to bootstrap a new Data Science project, hence the repo provides templates that can be reused to establish a cookie cutter approach for the bootstrapping process to shorten the process from days to hours or minutes. - -The best way to consume this accelerator will be to choose a complex use case that reflects most of your organisation’s needs from a Data Science perspective and start adjusting this accelerator to accommodate those requirements. The first use case may take longer to deliver, however, once the process has been ironed out, subsequent use cases can be onboarded in a matter of days if not hours. - -Following the demo helps to understand the concept of the solution accelerator, architectual pattern, and ongoing work extending the solution accelerator to other patterns. Feel free to replace the inner loop model with your model and rerun accordingly. - - -## 👤 Getting started: Azure Machine Learning - Classical Machine Learning Demo - -The demo follows the Classical Machine Learning Pattern with Azure Machine Learning. - -![AML CML](/documentation/architecturepattern/AzureML_CML_Architecture.png) +![AzureML CML](/documentation/architecturepattern/AzureML_CML_Architecture.png) ‼️ **Please follow the instructions to execute the demo accordingly: [Quickstart](https://github.com/Azure/mlops-v2/blob/main/QUICKSTART.md)** ‼️ -‼️ **Please hand in any issues here: [Issues](https://github.com/Azure/mlops-v2/issues)** ‼️ - +‼️ **Please submit any issues here: [Issues](https://github.com/Azure/mlops-v2/issues)** ‼️ ## 📐 Pattern Architectures: Key concepts | Link | AI Pattern | | ------------------------------------------------------- | ----------------------------------------------------------------------- | -| [Pattern AML CML](https://github.com/Azure/mlops-v2/blob/main/documentation/architecturepattern/AzureML_CML_Architecture.png) | Azure Machine Learning - Classical Machine Learning | -| [Pattern AML CV](https://github.com/Azure/mlops-v2/blob/main/documentation/architecturepattern/AzureML_SupervisedCV_Architecture.png) | Azure Machine Learning - Computer Vision | +| [Pattern AzureML CML](https://github.com/Azure/mlops-v2/blob/main/documentation/architecturepattern/AzureML_CML_Architecture.png) | Azure Machine Learning - Classical Machine Learning | +| [Pattern AzureML CV](https://github.com/Azure/mlops-v2/blob/main/documentation/architecturepattern/AzureML_SupervisedCV_Architecture.png) | Azure Machine Learning - Computer Vision | | [TBD] | Azure Machine Learning - Natural Language Processing | | [TBD] | Azure Machine Learning / Azure Databricks - Classical Machine Learning | | [TBD] | Azure Machine Learning / Azure Databricks - Computer Vision | | [TBD] | Azure Machine Learning / Azure Databricks - Natural Language Processing | | [TBD] | Azure Machine Learning - Edge AI | - -## 📯 (Coming Soon) One-click Deployments: +## 📯 (Coming Soon) One-click deployments ## 📯 MLOps infrastructure deployment @@ -82,7 +72,6 @@ The demo follows the Classical Machine Learning Pattern with Azure Machine Learn | [TBD] | Natural Language Processing | Azure Machine Learning, Azure Databricks | [DEPLOY BUTTON] | | [TBD] | Edge AI | Azure Machine Learning | [DEPLOY BUTTON] | - ## Contributing This project welcomes contributions and suggestions. To learn more visit the contributing section, see [CONTRIBUTING.md](CONTRIBUTING.md) for details. @@ -93,7 +82,6 @@ When you submit a pull request, a CLA bot will automatically determine whether y This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. - ## Trademarks This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft diff --git a/config-infra-dev.yml b/config-infra-dev.yml index 104bf7d..0939c6e 100644 --- a/config-infra-dev.yml +++ b/config-infra-dev.yml @@ -1,6 +1,3 @@ -# Copyright (c) Microsoft Corporation. All rights reserved. -# Licensed under the MIT License. - variables: # Global diff --git a/config-infra-prod.yml b/config-infra-prod.yml index 75715be..de957bd 100644 --- a/config-infra-prod.yml +++ b/config-infra-prod.yml @@ -1,6 +1,3 @@ -# Copyright (c) Microsoft Corporation. All rights reserved. -# Licensed under the MIT License. - variables: # Global diff --git a/data-science-regression/documentation/repositoryfiles/readme.md b/data-science-regression/documentation/repositoryfiles/readme.md deleted file mode 100644 index 8b13789..0000000 --- a/data-science-regression/documentation/repositoryfiles/readme.md +++ /dev/null @@ -1 +0,0 @@ - diff --git a/data-science/documentation/repositoryfiles/readme.md b/data-science/documentation/repositoryfiles/readme.md deleted file mode 100644 index 8b13789..0000000 --- a/data-science/documentation/repositoryfiles/readme.md +++ /dev/null @@ -1 +0,0 @@ - diff --git a/data-science/experimentation/experiment1.ipynb b/data-science/experimentation/experiment.ipynb similarity index 100% rename from data-science/experimentation/experiment1.ipynb rename to data-science/experimentation/experiment.ipynb diff --git a/documentation/architecturepattern/readme.md b/documentation/architecturepattern/readme.md deleted file mode 100644 index 8b13789..0000000 --- a/documentation/architecturepattern/readme.md +++ /dev/null @@ -1 +0,0 @@ - diff --git a/documentation/repositoryfiles/readme.md b/documentation/repositoryfiles/readme.md deleted file mode 100644 index 8b13789..0000000 --- a/documentation/repositoryfiles/readme.md +++ /dev/null @@ -1 +0,0 @@ - diff --git a/mlops/devops-pipelines/aml-cli-v2/deploy-batch-scoring-pipeline-v2.yml b/mlops/devops-pipelines/aml-cli-v2/deploy-batch-scoring-pipeline-v2.yml index 20c72d8..6e4fb3a 100644 --- a/mlops/devops-pipelines/aml-cli-v2/deploy-batch-scoring-pipeline-v2.yml +++ b/mlops/devops-pipelines/aml-cli-v2/deploy-batch-scoring-pipeline-v2.yml @@ -1,6 +1,3 @@ -# Copyright (c) Microsoft Corporation. All rights reserved. -# Licensed under the MIT License. - variables: - ${{ if eq(variables['Build.SourceBranchName'], 'main') }}: # 'main' branch: PRD environment diff --git a/mlops/devops-pipelines/aml-cli-v2/deploy-model-training-pipeline-v2.yml b/mlops/devops-pipelines/aml-cli-v2/deploy-model-training-pipeline-v2.yml index 430eb5d..f8700ff 100644 --- a/mlops/devops-pipelines/aml-cli-v2/deploy-model-training-pipeline-v2.yml +++ b/mlops/devops-pipelines/aml-cli-v2/deploy-model-training-pipeline-v2.yml @@ -1,6 +1,3 @@ -# Copyright (c) Microsoft Corporation. All rights reserved. -# Licensed under the MIT License. - variables: - ${{ if eq(variables['Build.SourceBranchName'], 'main') }}: # 'main' branch: PRD environment diff --git a/mlops/devops-pipelines/python-sdk/deploy-model-batch-scoring.yml b/mlops/devops-pipelines/python-sdk/deploy-model-batch-scoring.yml index c11c9cc..63a5626 100644 --- a/mlops/devops-pipelines/python-sdk/deploy-model-batch-scoring.yml +++ b/mlops/devops-pipelines/python-sdk/deploy-model-batch-scoring.yml @@ -1,6 +1,3 @@ -# Copyright (c) Microsoft Corporation. All rights reserved. -# Licensed under the MIT License. - variables: - template: ../../../config-aml.yml - ${{ if eq(variables['Build.SourceBranchName'], 'main') }}: diff --git a/mlops/devops-pipelines/python-sdk/deploy-model-training-pipeline.yml b/mlops/devops-pipelines/python-sdk/deploy-model-training-pipeline.yml index f750340..0755cea 100644 --- a/mlops/devops-pipelines/python-sdk/deploy-model-training-pipeline.yml +++ b/mlops/devops-pipelines/python-sdk/deploy-model-training-pipeline.yml @@ -1,6 +1,3 @@ -# Copyright (c) Microsoft Corporation. All rights reserved. -# Licensed under the MIT License. - variables: - template: ../../../config-aml.yml - ${{ if eq(variables['Build.SourceBranchName'], 'main') }}: diff --git a/mlops/environments/batch.yml b/mlops/environments/batch.yml index f8af984..27117e8 100644 --- a/mlops/environments/batch.yml +++ b/mlops/environments/batch.yml @@ -1,6 +1,3 @@ -# Copyright (c) Microsoft Corporation. All rights reserved. -# Licensed under the MIT License. - name: mnist-batch channels: - defaults