This workshop is archived and is no longer being maintained. Content is read-only.

For additional Data and AI content, please go to https://microsoft.github.io/sqlworkshops/.

Real-time data with Azure Database for PostgreSQL Hyperscale

Wide World Importers (WWI) is a traditional brick and mortar business with a long track record of success, generating profits through strong retail store sales of their unique offering of affordable products from around the world and a strong focus on customer relationships. WWI modernized their business by expanding to online storefronts and experimenting with various marketing tactics to drive online sales. This expansion has made it more challenging to analyze user clickstream data, online ad performance, and other marketing campaigns at scale, and to provide insights to the marketing team in real-time.

WWI is interested in learning to use advanced features of the managed PostgreSQL PaaS service on Azure to make their database more scalable and able to handle the rapid ingest of streaming data while simultaneously generating and serving pre-aggregated data for reports.

June 2020

Target audience

• Database Administrator
• Data Engineer
• Data Scientist
• Database Developer
• Solution Architect

Abstracts

Workshop

In this workshop, you will learn how to use advanced features of the managed PostgreSQL PaaS service on Azure to make your database more scalable and able to handle the rapid ingest of streaming data while simultaneously generating and serving pre-aggregated data for reports. You will design a resilient stream processing pipeline to ingest, process, and save real-time data to Postgres. Next, you will learn how to explore and gain insights from flat files at scale, perform aggregates, and save the valuable information to your database. Finally, you will create complex reports containing advanced visualizations, using a drag-and-drop interface, and use them to build a customizable dashboard that gets updated in real-time.

At the end of this workshop, you will be better able to implement a highly scalable, managed open source database solution that can simultaneously handle real-time data and roll-up and serve data for advanced visualizations.

Whiteboard design session

In this whiteboard design session, you will work with a group to design a solution for using advanced features of the managed PostgreSQL PaaS service on Azure to make your database more scalable and able to handle the rapid ingest of streaming data while simultaneously generating and serving pre-aggregated data for reports. You will provide guidance for designing a resilient stream processing pipeline to ingest, process, and save real-time data to Postgres. Next, you will design a solution for exploring and gaining insights from flat files at scale, perform aggregates, and saving the valuable information to your database. Finally, you will provide guidance on how to create complex reports containing advanced visualizations, using a drag-and-drop interface, and use them to build a customizable dashboard that gets updated in real-time.

At the end of this whiteboard design session, you will be better able to design a highly scalable, managed open source database solution that can simultaneously handle real-time data and roll-up and serve data for advanced visualizations.

Hands-on lab

In this hands-on lab, you will implement a proof-of-concept (PoC) for using advanced features of the managed PostgreSQL PaaS service on Azure. These features help make your database more scalable and able to handle the rapid ingest of streaming data while simultaneously generating and serving pre-aggregated data for reports. You will create a resilient stream processing pipeline to ingest, process, and save real-time data to Postgres. Next, you will implement a solution for exploring and gaining insights from flat files at scale, perform aggregates, and saving the valuable information to your database. Finally, you will create complex reports containing advanced visualizations, using a drag-and-drop interface, and use them to build a customizable dashboard that gets updated in real-time.

At the end of this hands-on-lab, you will be better able to implement a highly scalable, managed open source database solution that can simultaneously handle real-time data and roll-up and serve data for advanced visualizations.

Azure services and related products

• Azure Database for PostgreSQL
• Azure Event Hubs with Kafka
• Azure Databricks
• Power BI
• Azure Cloud Shell
• pgAdmin

Related references

• MCW