microsoft
/
devsquad-in-a-day
зеркало из https://github.com/microsoft/devsquad-dataops.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
DevSquad In a Day - DataOps. A hands-on lab for exercising DataOps practices on Azure.
448 коммитов 3 Ветки 0 Теги 54 MiB
PowerShell 57.4%
Python 42.2%
Gherkin 0.4%
Перейти к файлу
Fabio Padua 617103346c
Issue#52 - Documentation adjustments (#53) 
* Rebase (#3)

* chore: enviornments variables adjustment

* chore: schema update

* fix: add SP version on job variable

* chore: force powershell version

* chore: change task order to publish DB secrets

* chore: Adjustments on Databricks Secrets deploy

* chore: adjust variables info on parameter

* chore: adjustment on Databricks scripts

* chore: dataLakeName variable adjustment

* chore: dataComputeRS variable change

* chore: documentation adjustment att image

* chore: change keyvalt set permissions on SP

* chore: Databricks task order

* chore: databricks adjustments

* chore: var output display removal

* chore: databricks adjustments

* chore: documentation adjustment

* chore: documentation change

* chore: script adjustment

* chore: add command to list scope content

* chore: add SP into Admin role

* chore: adjust Administrator Role SP

* chore: add connect to AD first

* chore: Remove connection do Azure AD

* chore: documentation adjustment

* chore: fix uppercase extensions on some images (#42)

* chore: fix uppercase extensions on some images

* chore: directory changes

* chore: directory rename

* Issue 43 - Templates Changes (#44)

* chore: fix uppercase extensions on some images

* chore: directory changes

* chore: directory rename

* chore: Service Principal secrets first adjustments

* fix: adjustment of SP

* chore: save secret on output

* chore: syntax fix

* chore: token adjustment

* chore: store secret inside output hol file

* chore: put SP inside parameters

* chore: template adjustment

* chore: lower parameter for fix

* chore: syntax adjustment

* chore: Add logs

* chore: add log info

* chore: log info

* Issue #43 - Adjust the pipeline run to use first time SP secret (#45)

* chore: fix uppercase extensions on some images

* chore: directory changes

* chore: directory rename

* chore: Service Principal secrets first adjustments

* fix: adjustment of SP

* chore: save secret on output

* chore: syntax fix

* chore: token adjustment

* chore: store secret inside output hol file

* chore: put SP inside parameters

* chore: template adjustment

* chore: lower parameter for fix

* chore: syntax adjustment

* chore: Add logs

* chore: add log info

* chore: log info

* chore: pipeline execution change

* chore: documentation adjustments

* feat: Issue templates creations

- Bug Issue template
- New Feature request issue template

* Update issue templates

* Issue#35 - Change the store of sample-data to current repo (#46)

* Rebase (#2)

* chore: enviornments variables adjustment

* chore: schema update

* fix: add SP version on job variable

* chore: force powershell version

* chore: change task order to publish DB secrets

* chore: Adjustments on Databricks Secrets deploy

* chore: adjust variables info on parameter

* chore: adjustment on Databricks scripts

* chore: dataLakeName variable adjustment

* chore: dataComputeRS variable change

* chore: documentation adjustment att image

* chore: change keyvalt set permissions on SP

* chore: Databricks task order

* chore: databricks adjustments

* chore: var output display removal

* chore: databricks adjustments

* chore: documentation adjustment

* chore: documentation change

* chore: script adjustment

* chore: add command to list scope content

* chore: add SP into Admin role

* chore: adjust Administrator Role SP

* chore: add connect to AD first

* chore: Remove connection do Azure AD

* chore: documentation adjustment

* chore: fix uppercase extensions on some images (#42)

* chore: fix uppercase extensions on some images

* chore: directory changes

* chore: directory rename

* Issue 43 - Templates Changes (#44)

* chore: fix uppercase extensions on some images

* chore: directory changes

* chore: directory rename

* chore: Service Principal secrets first adjustments

* fix: adjustment of SP

* chore: save secret on output

* chore: syntax fix

* chore: token adjustment

* chore: store secret inside output hol file

* chore: put SP inside parameters

* chore: template adjustment

* chore: lower parameter for fix

* chore: syntax adjustment

* chore: Add logs

* chore: add log info

* chore: log info

* Issue #43 - Adjust the pipeline run to use first time SP secret (#45)

* chore: fix uppercase extensions on some images

* chore: directory changes

* chore: directory rename

* chore: Service Principal secrets first adjustments

* fix: adjustment of SP

* chore: save secret on output

* chore: syntax fix

* chore: token adjustment

* chore: store secret inside output hol file

* chore: put SP inside parameters

* chore: template adjustment

* chore: lower parameter for fix

* chore: syntax adjustment

* chore: Add logs

* chore: add log info

* chore: log info

* chore: pipeline execution change

* chore: documentation adjustments

* chore: sample-data upload to main-stream

* chore: unzip and upload sample-data

* fix: adjustment on pipe run

* Issue#47 - Change Databricks version (#48)

* Rebase (#2)

* chore: enviornments variables adjustment

* chore: schema update

* fix: add SP version on job variable

* chore: force powershell version

* chore: change task order to publish DB secrets

* chore: Adjustments on Databricks Secrets deploy

* chore: adjust variables info on parameter

* chore: adjustment on Databricks scripts

* chore: dataLakeName variable adjustment

* chore: dataComputeRS variable change

* chore: documentation adjustment att image

* chore: change keyvalt set permissions on SP

* chore: Databricks task order

* chore: databricks adjustments

* chore: var output display removal

* chore: databricks adjustments

* chore: documentation adjustment

* chore: documentation change

* chore: script adjustment

* chore: add command to list scope content

* chore: add SP into Admin role

* chore: adjust Administrator Role SP

* chore: add connect to AD first

* chore: Remove connection do Azure AD

* chore: documentation adjustment

* chore: fix uppercase extensions on some images (#42)

* chore: fix uppercase extensions on some images

* chore: directory changes

* chore: directory rename

* Issue 43 - Templates Changes (#44)

* chore: fix uppercase extensions on some images

* chore: directory changes

* chore: directory rename

* chore: Service Principal secrets first adjustments

* fix: adjustment of SP

* chore: save secret on output

* chore: syntax fix

* chore: token adjustment

* chore: store secret inside output hol file

* chore: put SP inside parameters

* chore: template adjustment

* chore: lower parameter for fix

* chore: syntax adjustment

* chore: Add logs

* chore: add log info

* chore: log info

* Issue #43 - Adjust the pipeline run to use first time SP secret (#45)

* chore: fix uppercase extensions on some images

* chore: directory changes

* chore: directory rename

* chore: Service Principal secrets first adjustments

* fix: adjustment of SP

* chore: save secret on output

* chore: syntax fix

* chore: token adjustment

* chore: store secret inside output hol file

* chore: put SP inside parameters

* chore: template adjustment

* chore: lower parameter for fix

* chore: syntax adjustment

* chore: Add logs

* chore: add log info

* chore: log info

* chore: pipeline execution change

* chore: documentation adjustments

* chore: sample-data upload to main-stream

* chore: unzip and upload sample-data

* fix: adjustment on pipe run

* chore: changed databrick version

* PowerShell version changed when run pipeline (#50)

* Rebase (#2)

* chore: enviornments variables adjustment

* chore: schema update

* fix: add SP version on job variable

* chore: force powershell version

* chore: change task order to publish DB secrets

* chore: Adjustments on Databricks Secrets deploy

* chore: adjust variables info on parameter

* chore: adjustment on Databricks scripts

* chore: dataLakeName variable adjustment

* chore: dataComputeRS variable change

* chore: documentation adjustment att image

* chore: change keyvalt set permissions on SP

* chore: Databricks task order

* chore: databricks adjustments

* chore: var output display removal

* chore: databricks adjustments

* chore: documentation adjustment

* chore: documentation change

* chore: script adjustment

* chore: add command to list scope content

* chore: add SP into Admin role

* chore: adjust Administrator Role SP

* chore: add connect to AD first

* chore: Remove connection do Azure AD

* chore: documentation adjustment

* chore: fix uppercase extensions on some images (#42)

* chore: fix uppercase extensions on some images

* chore: directory changes

* chore: directory rename

* Issue 43 - Templates Changes (#44)

* chore: fix uppercase extensions on some images

* chore: directory changes

* chore: directory rename

* chore: Service Principal secrets first adjustments

* fix: adjustment of SP

* chore: save secret on output

* chore: syntax fix

* chore: token adjustment

* chore: store secret inside output hol file

* chore: put SP inside parameters

* chore: template adjustment

* chore: lower parameter for fix

* chore: syntax adjustment

* chore: Add logs

* chore: add log info

* chore: log info

* Issue #43 - Adjust the pipeline run to use first time SP secret (#45)

* chore: fix uppercase extensions on some images

* chore: directory changes

* chore: directory rename

* chore: Service Principal secrets first adjustments

* fix: adjustment of SP

* chore: save secret on output

* chore: syntax fix

* chore: token adjustment

* chore: store secret inside output hol file

* chore: put SP inside parameters

* chore: template adjustment

* chore: lower parameter for fix

* chore: syntax adjustment

* chore: Add logs

* chore: add log info

* chore: log info

* chore: pipeline execution change

* chore: documentation adjustments

* chore: sample-data upload to main-stream

* chore: unzip and upload sample-data

* fix: adjustment on pipe run

* chore: changed databrick version

* chore: changes into PS Version on Pipe execution

* [Fix] - Remove logs (#51)

* Rebase (#2)

* chore: enviornments variables adjustment

* chore: schema update

* fix: add SP version on job variable

* chore: force powershell version

* chore: change task order to publish DB secrets

* chore: Adjustments on Databricks Secrets deploy

* chore: adjust variables info on parameter

* chore: adjustment on Databricks scripts

* chore: dataLakeName variable adjustment

* chore: dataComputeRS variable change

* chore: documentation adjustment att image

* chore: change keyvalt set permissions on SP

* chore: Databricks task order

* chore: databricks adjustments

* chore: var output display removal

* chore: databricks adjustments

* chore: documentation adjustment

* chore: documentation change

* chore: script adjustment

* chore: add command to list scope content

* chore: add SP into Admin role

* chore: adjust Administrator Role SP

* chore: add connect to AD first

* chore: Remove connection do Azure AD

* chore: documentation adjustment

* chore: fix uppercase extensions on some images (#42)

* chore: fix uppercase extensions on some images

* chore: directory changes

* chore: directory rename

* Issue 43 - Templates Changes (#44)

* chore: fix uppercase extensions on some images

* chore: directory changes

* chore: directory rename

* chore: Service Principal secrets first adjustments

* fix: adjustment of SP

* chore: save secret on output

* chore: syntax fix

* chore: token adjustment

* chore: store secret inside output hol file

* chore: put SP inside parameters

* chore: template adjustment

* chore: lower parameter for fix

* chore: syntax adjustment

* chore: Add logs

* chore: add log info

* chore: log info

* Issue #43 - Adjust the pipeline run to use first time SP secret (#45)

* chore: fix uppercase extensions on some images

* chore: directory changes

* chore: directory rename

* chore: Service Principal secrets first adjustments

* fix: adjustment of SP

* chore: save secret on output

* chore: syntax fix

* chore: token adjustment

* chore: store secret inside output hol file

* chore: put SP inside parameters

* chore: template adjustment

* chore: lower parameter for fix

* chore: syntax adjustment

* chore: Add logs

* chore: add log info

* chore: log info

* chore: pipeline execution change

* chore: documentation adjustments

* chore: sample-data upload to main-stream

* chore: unzip and upload sample-data

* fix: adjustment on pipe run

* chore: changed databrick version

* chore: changes into PS Version on Pipe execution

* chore: tests adjustments

* chore: log adjustments

* chore: documentation change

* chore: documentation adjustments
 		2022-02-24 10:07:50 -03:00
.github/ISSUE_TEMPLATE Update issue templates 2022-02-16 09:12:33 -03:00
azure-pipelines PowerShell version changed when run pipeline (#50) 2022-02-18 09:44:17 -03:00
data-platform chore: fix uppercase extensions on some images (#42) 2022-02-14 17:12:06 -03:00
hands-on-lab Issue#52 - Documentation adjustments (#53) 2022-02-24 10:07:50 -03:00
infrastructure-as-code [Fix] - Remove logs (#51) 2022-02-22 11:34:32 -03:00
quickstart [Fix] - Remove logs (#51) 2022-02-22 11:34:32 -03:00
.gitignore refactor: changed folders structure 2021-09-22 10:34:51 -03:00
CODE_OF_CONDUCT.md docs: open source docs 2021-09-22 10:33:21 -03:00
CONTRIBUTING.md docs: open source docs 2021-09-22 10:33:21 -03:00
LICENSE docs: Added MIT LICENSE 2021-09-22 13:26:48 -03:00
README.md chore: documentation adjustment att image 2022-02-07 15:25:20 -03:00
SECURITY.md docs: open source docs 2021-09-22 10:33:21 -03:00

README.md

DevSquad In a Day

Margie's Travel (MT) provides concierge services for business travelers. In an increasingly crowded market, they are always looking for ways to differentiate themselves and provide added value to their corporate customers.

Recently they've developed a POC for a web app that their internal customer service agents can use to provide additional valuable information to the traveler during the flight booking process. On that, POC they've enabled their agents to enter in the flight information and produce a prediction as to whether the departing flight will encounter a 15-minute or longer delay, considering the weather forecast for the departure hour. Now they want to evaluate deploy the project to production, leveraging DataOps & Software Engineering best practices.

Sept 2021

Target audience

  • Software Engineers
  • Data Engineers
  • Data Architects

Abstracts

Workshop

In this workshop, you will deploy a DataOps reference arquitecture, for understanding best practices of Data Engineering & Software Engineering combined.

'Solution Architecture'

Lab Instructions

  1. Follow these quickstart instructions to setup your lab environment before starting the Hands-On Lab.

  2. Follow the Hands-On Lab instructions.

Azure services and related products

  • Azure DevOps
  • Azure Databricks
  • Azure Data Factory
  • Azure Data Lake Storage Gen2

Help & Support

We welcome feedback and comments from Microsoft SMEs & learning partners who deliver this workshop.

Having trouble?

  • First, verify you have followed all written lab instructions (including the quickstart documents).
  • Next, submit an issue with a detailed description of the problem.
  • Do not submit pull requests. Our content authors will make all changes and submit pull requests for approval.

If you are planning to present a workshop, review and test the materials early! We recommend at least two weeks prior.

Please allow 5 - 10 business days for review and resolution of issues.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.