06b4e5597c
* Created WhatTheHack template stub * Updated Readme * Updated README * updated README * Update README.md * Update README.md * updated README * Updated README * updated README * updated png * Updated png * updated png * updated README * updated README * Updated Readme * Updated README * Updated * updated title * Updated Title * Updated Title * Update Solution-04.md * Updated TItle * Updated Title * Update Challenge-03.md * Update Challenge-04.md * Update README.md * Updated Coaches README * Updated Coach README * Updated Challenge * Updated Solution and Challenge * Updated Solution * Updated multiple items * Updated Multiple items * Updated README * Update README.md * Updated Challenge * updated Challenge * updated Challenge * Updated Solution * updated Solution * updated Solution * Updated Solution * Updated Folder Structure * updated Solution * updated Solution * FA: Testing out the commits * FA: Testing the commits from VSCode * FA: Adding more content * FA: Adding more content * FA: Fixing formatting * FA: Formatting * FA: Formatting * FA: Formatting * FA: Updated formatting * FA: Formatting and Spell Check * FA: Formatting and Spell Check * FA: Adding more points for Challenge 02 * FA: Formatting and Spell Check * FA: Formatting and Spell Check * FA: Adding more content * FA: Adding more content * FA: Adding more content * Adding new images * FA: Adding images * Adding images * FA: Adding more content * FA: Formatting * FA: Formatting * FA: Additonal formatting * FA: Formatting * FA: Formatting updated * FA: Uploading image * Adding pictures and links * FA: Adding new image * FA: Adding new image * FA: Adding new learning resources * FA: Added Challenge 4 image * FA: Adding more content to the Challenge * FA: Formatting * FA: Adding more points to C3 and C4 * FA: Updating the links and formatting * FA: Updating links and formatting * FA: Adding example solutions * Delete Challenge 4 Example.png * FA: Adding example solution * Delete Challenge4_ Example.png * Add files via upload * FA: Addinng more content * FA: Adding example solutions * Delete Challenge3a_Example.png * Delete Challenge3b_Example.png * FA: Adding new solution files * FA: Adding more details to the solution * FA: Spell check * FA: Added more links to the solutions * FA: Spell check * FA: Spell check * FA: Adding more details on the Bonus Challenge * FA: Adding more details on the Bonus Challenge * FA: Formatting updates * FA: Updating formatting * FA: Fixed section on data formats * FA: Fixed section on data formats * Updated Challenges * Updated Challenge * FA: Updating images * Updated PowerPoint * Updated PowerPoint * Updated PowerPoint * Updated Challenge * Updated Challenge * Updated Challenge * Updated PowerPoint * FA: Added Delta lake Notebook * Updated Challenge * Updated Challenge * FA: Fixing typo * Updated Challenge * Updated Solution * Updated Challenge * Updated PowerPoint * Updated Challenges * updated Challenge * Updated Challenge * Updated Challenge * Updated Challenge * Updated Challenge * Updated Challenge and Solution * Updated Challenge * updated Challenge * Updated Challenge * FA: Adding more details about Datamarts and Azure Serverless * Updated Solution * FA: Updated, adding advice on sharing access to RGs * FA: Adding Business Case details * FA: Adding addtional Business Case details * FA: Added more content and Business Cases * updated Challenge * Updated Challenge * Updated PowerPoint * Updated Challenges * Updated Solutions * Updated Solution * updated Challenges and Solutions * updated README * Updated README * Update Challenge-00.md * Update Challenge-02.md * Update Challenge-03.md * Update README.md Fixed some typos * Update Solution-02.md Removed an extra space * Update Solution-03.md Added a missing comma * Update README.md Fixed some capitalization for Power BI * Update README.md Removed periods in bullet list to be consistent * Update README.md Removed an extra period * Update Challenge-00.md Added a missing question mark * Update Challenge-01.md changed a semi-colon to a colon * Update Challenge-02.md Fixed a typo * Updated Coaches Guide Descriptions * Updated Coach Solution Added the SQL Sample database links. * Updated Coaches Solution * Updated Coaches Solution * Update Challenge-01.md added spacing to break up big paragraph. * Update Challenge-01.md * Update Challenge-01.md * Update Challenge-02.md * Update Challenge-03.md * Updated Student Challenge * Updated Student Challenge 1 * Updated Student Challenge 1 * Updated Student Challenge 1 * Updated Student Challenge 1 * Updated Student Challenge 1 * Updated Student Challenge 1 * Updated Student Challenge 1 * Updated Student Challenge 1 * Updated Student Challenge 1 * FA - Updating based on Peter's review * FA: Updated based on Peter's feedbacl * Updated Coaches Readme Added Key Vault information. * Update README.md minor updates to refer to students instead of "users". * Update README.md code ticks on script file name. * Update Solution-01.md * Update Solution-01.md --------- Co-authored-by: GitHub Actions Bot <> Co-authored-by: FarhanArif <farhanarif@microsoft.com> Co-authored-by: Andy Huang <54148527+Whowong@users.noreply.github.com> Co-authored-by: Pete Rodriguez <perktime@users.noreply.github.com> Co-authored-by: Peter C. Laudati <plaudati@hotmail.com> |
||
|---|---|---|
| .. | ||
| Coach | ||
| Student | ||
| img | ||
| README.md | ||
README.md
What The Hack | Bronze-Silver-Gold Using Synapse and Databricks
Introduction
With many customers moving towards a modern three-tiered Data Lake architecture it is imperative that we understand how to utilize Synapase and Databricks to build out the bronze, silver and gold layers to serve data to Power BI for dashboards and reporting while also ensuring that the bronze and silver layers are being hydrated correctly for ML/AI workloads.
Learning Objectives
In this hack you will be solving the common big data architecture issue of bringing multiple data sources together, standardizing them in a Delta format, and serving them up to Power BI for dashboards and reporting. You will get hands on with Azure Synapse and Databricks so that you understand the similarities and differences between them and how they both can collaborate on the same data, including Delta technology.
Challenges
- Challenge 00: Prerequisites - Ready, Set, GO!
- Understand the basics of Synapse
- Understand the basics of Databricks
- Understand Delta Lake Concepts
- Understand the basics of Key Vault
- Understand Data Lake Best Practices
- Challenge 01: Building out the Bronze
- Standup and configure the Synapse and Databricks Environments
- Agree on and begin to implement the three-tiered architecture
- Hydrate the Bronze Data Lake
- Ensure no connection details are stored on the Linked Service or in Notebooks
- Challenge 02: Standardizing on Silver
- Move the data from the Bronze Layer to the Silver Layer
- Apply Delta Format to the Silver Layer
- Perform consolidation and standardization
- Challenge 03: Go for the Gold
- Take data from the Silver Layer and make it business analyst ready
- Understand the basics of data models
- Challenge 04: Visualize the Results
- Create Power BI assets to showcase your results
Prerequisites
- Your own Azure subscription with Owner access
- Willingness to get your hands dirty and learn from your colleagues
- A wicked cool Teams background. There will be prizes for each day