464 строки
24 KiB
Markdown
464 строки
24 KiB
Markdown
|
# Lab 6 – Automating with Synapse Pipelines
|
|||
|
|
|||
|
<img src="images/lab06/media/image1.png" style="width:3.9375in;height:0.63819in" />
|
|||
|
<br/>
|
|||
|
<img src="images/lab06/media/image2.png" style="width:3.48973in;height:1.20479in" />
|
|||
|
|
|||
|
#
|
|||
|
|
|||
|
Contents
|
|||
|
|
|||
|
#
|
|||
|
|
|||
|
[Lab Overview](#lab-overview)
|
|||
|
|
|||
|
[Introduction](#introduction)
|
|||
|
|
|||
|
[Objectives](#objectives)
|
|||
|
|
|||
|
[Exercise 1: Orchestrate the refresh and transformation of the customer
|
|||
|
charges
|
|||
|
file](#exercise-1-orchestrate-the-refresh-and-transformation-of-the-customer-charges-file)
|
|||
|
|
|||
|
- [Step 1: Open Synapse Studio in a web browser](#step-1-open-synapse-studio-in-a-web-browser)
|
|||
|
|
|||
|
- [Step 2: Download the sample charge detail file](#step-2-download-the-sample-charge-detail-file)
|
|||
|
|
|||
|
- [Step 3: Automate the data integration and transformation for customer charges](#step-3-automate-the-data-integration-and-transformation-for-customer-charges)
|
|||
|
|
|||
|
[Exercise 2: Automate the rerun of your machine learning
|
|||
|
pipeline](#exercise-2-automate-the-rerun-of-your-machine-learning-pipeline)
|
|||
|
|
|||
|
- [Step 1: Setup a Synapse pipeline which runs your machine learning pipeline](#step-1-setup-a-synapse-pipeline-which-runs-your-machine-learning-pipeline)
|
|||
|
|
|||
|
- [Step 2: Schedule the Synapse pipeline](#step-2-schedule-the-synapse-pipeline)
|
|||
|
|
|||
|
- [Step 3: Schedule the data source refresh in Customer Insights](#step-3-schedule-the-data-source-refresh-in-customer-insights)
|
|||
|
|
|||
|
[Exercise 3: Cleanup Azure resources (optional)](#exercise-3-cleanup-azure-resources-optional)
|
|||
|
|
|||
|
[Summary](#summary)
|
|||
|
|
|||
|
#
|
|||
|
|
|||
|
# Lab Overview
|
|||
|
|
|||
|
## Introduction
|
|||
|
|
|||
|
This hands on lab will automate the refresh and transformation of a key
|
|||
|
customer charges file with Synapse pipelines and Mapping data flows. It
|
|||
|
will then automate the execution of the Azure ML pipeline using Synapse
|
|||
|
pipelines and how to automate the refresh of data into Customer
|
|||
|
Insights.
|
|||
|
|
|||
|
## Objectives
|
|||
|
|
|||
|
The objectives of this exercise are to:
|
|||
|
|
|||
|
- Learn how to utilize Synapse pipelines and Mapping data flows to do
|
|||
|
data integration and transformations with a drag-and-drop interface
|
|||
|
|
|||
|
- Learn how Synapse pipelines can schedule a machine learning to rerun
|
|||
|
|
|||
|
- Learn how to schedule the refresh of data into Customer Insights
|
|||
|
|
|||
|
The estimated time for this lab is 45 minutes
|
|||
|
|
|||
|
# Exercise 1: Orchestrate the refresh and transformation of the customer charges file
|
|||
|
|
|||
|
In this section you will perform some data integration and data
|
|||
|
transformation tasks to refresh the customer charges summary file from a
|
|||
|
more detailed file. You will use Synapse pipelines and Mapping data
|
|||
|
flows to accomplish this.
|
|||
|
|
|||
|
## Step 1: Open Synapse Studio in a web browser
|
|||
|
|
|||
|
Go to <https://portal.azure.com> and sign in with your organizational
|
|||
|
account.
|
|||
|
|
|||
|
In the search box at the top of the portal, search for “asaworkspace”
|
|||
|
and click on the Synapse workspace (not the SQL Server) which appears
|
|||
|
under the Resources section.
|
|||
|
|
|||
|
<img src="images/lab06/media/image3.png" style="width:4.90693in;height:4.15683in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
On the Overview blade and the Essentials section, click the Workspace
|
|||
|
web URL link to open Synapse Studio.
|
|||
|
|
|||
|
## Step 2: Download the sample charge detail file
|
|||
|
|
|||
|
1. Right click on this
|
|||
|
[CustomerChargesDetail.csv](https://raw.githubusercontent.com/ArtisConsulting/customer-insights-azure-data-workshop/main/SampleData/CustomerChargesDetail.csv)
|
|||
|
link and choose “Save link as…” and name the file
|
|||
|
**CustomerChargesDetail.csv** (not CustomerChargesDetail.txt) on your workstation. We will now upload this
|
|||
|
file to Azure Data Lake Storage Gen2 (ADLS).
|
|||
|
|
|||
|
1. Click on the Data pane on the left. It is the
|
|||
|
<img src="images/lab06/media/image4.png" style="width:0.28129in;height:0.36463in" />
|
|||
|
icon. Then go to the Linked tab. Expand Azure Data Lake Storage Gen2 and
|
|||
|
click on the default storage account (asaworkspace\<suffix>) and click
|
|||
|
on the staging container.
|
|||
|
|
|||
|
<img src="images/lab06/media/image5.png" style="width:3.50049in;height:3.6776in" alt="Graphical user interface, text, application Description automatically generated" />
|
|||
|
|
|||
|
1. Click the “+ New folder” button and create a new folder called
|
|||
|
**ChargesDetail**.
|
|||
|
|
|||
|
<img src="images/lab06/media/image6.png" style="width:7.5in;height:2.05208in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. Double click the new ChargesDetail folder. Then click the Upload button:
|
|||
|
|
|||
|
<img src="images/lab06/media/image7.png" />
|
|||
|
|
|||
|
1. Choose the CustomerChargesDetail.csv file you previously downloaded and
|
|||
|
click Upload.
|
|||
|
|
|||
|
<img src="images/lab06/media/image8.png" style="width:6.3238in;height:3.06293in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
For sake of this hands-on-lab, we will pretend that this
|
|||
|
CustomerChargesDetail.csv file is being updated and uploaded to this
|
|||
|
location daily by your accounting system.
|
|||
|
|
|||
|
## Step 3: Automate the data integration and transformation for customer charges
|
|||
|
|
|||
|
Let’s create a Synapse pipeline which takes the
|
|||
|
CustomerChargesDetail.csv file and transforms it using a Mapping data
|
|||
|
flow into the CustomerCharges.csv file we use in other labs. Mapping
|
|||
|
data flows are a visually-designed data transformations. A mapping data
|
|||
|
flow is executed under the covers on highly scalable Spark compute.
|
|||
|
|
|||
|
1. Go to the Integrate tab (the pipeline icon) in Synapse Studio:
|
|||
|
|
|||
|
<img src="images/lab06/media/image9.png" style="width:3.89638in;height:3.75052in" alt="Graphical user interface, application, Word Description automatically generated" />
|
|||
|
|
|||
|
1. Click the + sign and choose Pipeline to create a new Synapse pipeline.
|
|||
|
|
|||
|
<img src="images/lab06/media/image10.png" style="width:4.79234in;height:2.36491in" alt="Graphical user interface, text, application, Word Description automatically generated" />
|
|||
|
|
|||
|
1. In the Properties pane on the right, rename the pipeline and provide a
|
|||
|
description:
|
|||
|
|
|||
|
<img src="images/lab06/media/image11.png" style="width:3.03167in;height:2.36491in" alt="Graphical user interface, text, application Description automatically generated" />
|
|||
|
|
|||
|
1. In the Activities pane search for “data flow” and drag the
|
|||
|
**Data flow** module onto the canvas.
|
|||
|
|
|||
|
<img src="images/lab06/media/image12.png" style="width:7.31352in;height:2.48993in" alt="Graphical user interface, application Description automatically generated" />
|
|||
|
|
|||
|
1. With the Data flow module selected, view the General tab at the bottom
|
|||
|
of the screen and rename the module:
|
|||
|
|
|||
|
<img src="images/lab06/media/image13.png" style="width:5.82373in;height:1.85443in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. On the Settings tab, click the +New button to create a new Mapping data
|
|||
|
flow:
|
|||
|
|
|||
|
<img src="images/lab06/media/image14.png" style="width:5.66746in;height:2.16697in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. In the properties pane rename to “dfRefreshCustomerCharges” (“df” stands
|
|||
|
for data flow):
|
|||
|
|
|||
|
<img src="images/lab06/media/image15.png" style="width:3.06293in;height:2.48993in" alt="Graphical user interface, text, application Description automatically generated" />
|
|||
|
|
|||
|
1. Click the “Add source” box:
|
|||
|
|
|||
|
<img src="images/lab06/media/image16.png" style="width:3.93805in;height:1.97944in" alt="Application Description automatically generated with medium confidence" />
|
|||
|
|
|||
|
1. In the “Source settings” tab at the bottom name it
|
|||
|
**CustomerChargesDetail**, change to an **inline** source type, choose a
|
|||
|
**DelimitedText** dataset type, and choose the
|
|||
|
“**asaworkspace\<suffix>-WorkspaceDefaultStorage**” linked service
|
|||
|
(which was created automatically when the Synapse workspace was
|
|||
|
created).
|
|||
|
|
|||
|
<img src="images/lab06/media/image17.png" style="width:7.5in;height:3.22153in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. On the “Source options” tab click the Browse button on the File path:
|
|||
|
|
|||
|
<img src="images/lab06/media/image18.png" style="width:7.5in;height:1.67292in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. Browse to staging/ChargesDetail and select CustomerChargesDetail.csv and
|
|||
|
click OK:
|
|||
|
|
|||
|
<img src="images/lab06/media/image19.png" style="width:3.27129in;height:1.68774in" alt="Graphical user interface, application Description automatically generated" />
|
|||
|
|
|||
|
1. Check the “first row as header” checkbox:
|
|||
|
|
|||
|
<img src="images/lab06/media/image20.png" style="width:7.5in;height:4.96111in" alt="Graphical user interface, application Description automatically generated" />
|
|||
|
|
|||
|
1. On the Projection tab, notice the “Import schema” button is grayed out.
|
|||
|
Click the “Data flow debug” toggle at the top of the mapping data flow
|
|||
|
in order to spin up debug compute to assist in developing and debugging
|
|||
|
this mapping data flow.
|
|||
|
|
|||
|
<img src="images/lab06/media/image21.png" style="width:6.69885in;height:3.77136in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. Accept the defaults and click OK. The debug time to live of 1 hour means
|
|||
|
that the compute will auto-pause after 1 hour of inactivity in order to
|
|||
|
reduce your Azure cost.
|
|||
|
|
|||
|
<img src="images/lab06/media/image22.png" style="width:3.07335in;height:3.50049in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. Wait about 2-5 minutes until the data flow debug compute spins up. When
|
|||
|
complete you will get a notification in the top right and there will be
|
|||
|
a green checkmark next to the Data flow debug toggle. If you step away
|
|||
|
for more than an hour, the toggle debug compute will auto-pause and you
|
|||
|
will have to click the toggle again manually.
|
|||
|
|
|||
|
<img src="images/lab06/media/image23.png" style="width:3.63592in;height:0.80219in" alt="A picture containing text Description automatically generated" />
|
|||
|
|
|||
|
<img src="images/lab06/media/image24.png" style="width:1.86484in;height:0.30213in" />
|
|||
|
|
|||
|
1. Click the Import schema button (which was grayed out in step 13) now. Accepting the default formats is
|
|||
|
adequate for our file. Click the Import button at the bottom to accept the defaults:
|
|||
|
|
|||
|
<img src="images/lab06/media/image25.png" style="width:6.40714in;height:5.01112in" alt="Table Description automatically generated" />
|
|||
|
|
|||
|
1. Wait a few seconds for the schema to be loaded. Now the schema for this CustomerChargesDetail source is populated.
|
|||
|
|
|||
|
<img src="images/lab06/media/image26.png" style="width:7.5in;height:2.86389in" alt="Graphical user interface, application Description automatically generated" />
|
|||
|
|
|||
|
1. Go to the **Data preview** tab and click the **Refresh** button to see a
|
|||
|
preview of a few rows of this dataset.
|
|||
|
|
|||
|
<img src="images/lab06/media/image27.png" style="width:7.5in;height:2.24792in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. This file has one row per service per customer per month. We want to
|
|||
|
perform two data transformation tasks. First, we want to sum together
|
|||
|
the Charge and Tax columns. Second, we want to summarize all the
|
|||
|
services together into a single total charge per customer per
|
|||
|
ChargeDate.
|
|||
|
|
|||
|
Click the + sign next to the CustomerChargesDetail source and choose
|
|||
|
Derived Column
|
|||
|
|
|||
|
<img src="images/lab06/media/image28.png" style="width:5.0007in;height:5.90707in" alt="Graphical user interface, application Description automatically generated" />
|
|||
|
|
|||
|
1. Describe what you are doing in the Output stream name, set the column
|
|||
|
name to TotalCharges, and set the expression to Charge + Tax (you can
|
|||
|
click the “Open expression builder” button for help with building more
|
|||
|
complex expressions.
|
|||
|
|
|||
|
<img src="images/lab06/media/image29.png" style="width:7.5in;height:2.08056in" alt="Graphical user interface, text, application Description automatically generated" />
|
|||
|
|
|||
|
1. Click the + sign next to the SumChargesAndTax transform and add
|
|||
|
Aggregate:
|
|||
|
|
|||
|
<img src="images/lab06/media/image30.png" style="width:7.5in;height:4.85347in" alt="Graphical user interface Description automatically generated with medium confidence" />
|
|||
|
|
|||
|
1. Name the output stream AggregateByCustomerMonth (there is only one
|
|||
|
ChargeDate per month in the file), add customerID and ChargeDate
|
|||
|
(clicking the + sign next to customerID to add the second column to the
|
|||
|
Group by section:
|
|||
|
|
|||
|
<img src="images/lab06/media/image31.png" style="width:7.5in;height:5.4875in" alt="Graphical user interface Description automatically generated" />
|
|||
|
|
|||
|
1. Then click the “Aggregates” selector and add a TotalCharges column with
|
|||
|
expression sum(TotalCharges) which will aggregate the TotalCharges
|
|||
|
derived column to the per customer/ChargeDate grain.
|
|||
|
|
|||
|
<img src="images/lab06/media/image32.png" style="width:7.5in;height:3.18681in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. Click the + sign next to the AggregateByCustomerMonth and choose
|
|||
|
**Select**:
|
|||
|
|
|||
|
<img src="images/lab06/media/image33.png" style="width:5.19864in;height:4.6569in" alt="Graphical user interface, application Description automatically generated" />
|
|||
|
|
|||
|
1. The column order looks like this to begin with:
|
|||
|
|
|||
|
<img src="images/lab06/media/image34.png" style="width:7.5in;height:1.19931in" alt="Graphical user interface, application Description automatically generated" />
|
|||
|
|
|||
|
1. Rename the TotalCharges column to just Charge and mouse over that row
|
|||
|
until you see the six dots to the left of the row and drag it to reorder
|
|||
|
the columns. The columns should be in the order customerID, Charge,
|
|||
|
ChargeDate since this is the file format expected downstream.
|
|||
|
|
|||
|
<img src="images/lab06/media/image35.png" style="width:7.5in;height:1.11319in" alt="Graphical user interface, application Description automatically generated" />
|
|||
|
|
|||
|
1. Go to the Data preview tab on the Select1 module and click Refresh.
|
|||
|
Validate that the columns are in the order matching this screenshot and
|
|||
|
the column names match exactly:
|
|||
|
|
|||
|
<img src="images/lab06/media/image36.png" style="width:7.34477in;height:3.7922in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. Now let’s write out the results. Click the + icon next to the Select1
|
|||
|
module and search for “sink” and choose Sink:
|
|||
|
|
|||
|
<img src="images/lab06/media/image37.png" style="width:4.93819in;height:2.13572in" alt="A picture containing funnel chart Description automatically generated" />
|
|||
|
|
|||
|
1. Name the output stream CustomerCharges, choose an Inline sink type,
|
|||
|
choose a DelimitedText dataset type, choose the
|
|||
|
“asaworkspace\<suffix>-WorkspaceDefaultStorage” linked service:
|
|||
|
|
|||
|
<img src="images/lab06/media/image38.png" style="width:7.5in;height:3.20972in" alt="Graphical user interface, text, application Description automatically generated" />
|
|||
|
|
|||
|
1. Choose the staging container and the Charges folder. Check the “first
|
|||
|
row as header” checkbox and the “Clear the folder” checkbox:
|
|||
|
|
|||
|
<img src="images/lab06/media/image39.png" style="width:6.31338in;height:5.51119in" alt="Graphical user interface, application Description automatically generated" />
|
|||
|
|
|||
|
1. On the Optimize tab of the CustomerCharges sink you can keep the current
|
|||
|
“use current partitioning” setting as this will output multiple files in
|
|||
|
a highly parallel manner and generally perform better. The downstream
|
|||
|
Synapse Serverless SQL view called dbo.CustomerChurnCharges has been
|
|||
|
coded to read all files in the folder rather than looking for a specific
|
|||
|
filename. (If your scenario requires a single filename then choose “Single
|
|||
|
partition” but realize that will hurt performance.)
|
|||
|
|
|||
|
<img src="images/lab06/media/image40.png" style="width:6.49049in;height:1.05223in" alt="Graphical user interface, text, application, Word Description automatically generated" />
|
|||
|
|
|||
|
1. Flip back to the RefreshCustomerCharges pipeline tab, click the Debug
|
|||
|
dropdown and choose “Use data flow debug session”:
|
|||
|
|
|||
|
<img src="images/lab06/media/image41.png" style="width:7.5in;height:2.41181in" alt="Graphical user interface, text, application, chat or text message Description automatically generated" />
|
|||
|
|
|||
|
1. Go to the Data left nav, the Linked tab, choose the primary storage
|
|||
|
account, choose the staging container, browse to the Charges folder and
|
|||
|
you should see about 200 CSV files which are about 38KB each (compared
|
|||
|
to the original 7MB CustomerCharges.csv file):
|
|||
|
|
|||
|
<img src="images/lab06/media/image42.png" style="width:7.5in;height:2.71875in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. Click the Publish all button to save the pipeline and mapping data flow
|
|||
|
and make them available to schedule or for other users to see.
|
|||
|
|
|||
|
<img src="images/lab06/media/image43.png" style="width:4.1985in;height:0.38547in" />
|
|||
|
|
|||
|
1. Confirm the list of objects to publish and click the Publish button at
|
|||
|
the bottom:
|
|||
|
|
|||
|
<img src="images/lab06/media/image44.png" style="width:5.55286in;height:2.77122in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
# Exercise 2: Automate the rerun of your machine learning pipeline
|
|||
|
|
|||
|
In this section you will automate the execution of your machine learning
|
|||
|
pipeline. There are several ways to automate and operationalize your
|
|||
|
model. One option is to publish a batch inference pipeline which reuses
|
|||
|
a trained model to make more predictions (inferences). A second option is
|
|||
|
to automate the execution of the entire pipeline which retrains models
|
|||
|
and scores the dataset with each execution. We will use this
|
|||
|
second option in this lab.
|
|||
|
|
|||
|
## Step 1: Setup a Synapse pipeline which runs your machine learning pipeline
|
|||
|
|
|||
|
1. On the Integrate left nav, open the RefreshCustomerChargesAndChurnML
|
|||
|
pipeline, search for “machine” in the Activities panel, and drag
|
|||
|
**Machine Learning Execute Pipeline** onto the canvas:
|
|||
|
|
|||
|
<img src="images/lab06/media/image45.png" style="width:7.5in;height:2.02222in" alt="Graphical user interface Description automatically generated" />
|
|||
|
|
|||
|
1. Drag the green square from the Refresh Customer Charges data flow module
|
|||
|
to the Machine learning module. This sets a dependency to ensure the
|
|||
|
refresh of the charges file completes successfully before the machine
|
|||
|
learning pipeline is run:
|
|||
|
|
|||
|
<img src="images/lab06/media/image46.png" style="width:4.92777in;height:1.10432in" alt="A picture containing waterfall chart Description automatically generated" />
|
|||
|
|
|||
|
1. Click on the ML module and rename it Execute Churn ML Pipeline:
|
|||
|
|
|||
|
<img src="images/lab06/media/image47.png" style="width:5.91749in;height:1.12516in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. On the Settings tab for the ML module, click the new button to create a
|
|||
|
new Azure ML linked service:
|
|||
|
|
|||
|
<img src="images/lab06/media/image48.png" style="width:5.95916in;height:1.03139in" alt="Graphical user interface, text, application Description automatically generated" />
|
|||
|
|
|||
|
1. Name the linked service “lsAzureML” (the “ls” prefix stands for Linked
|
|||
|
Service), ensure the authentication is the (Synapse) Managed Identity
|
|||
|
(which was granted RBAC Contributor role on the Azure ML workspace in
|
|||
|
the ARM template), then choose the right Azure subscription and the
|
|||
|
“amlworkspace\<suffix>” Azure ML workspace.
|
|||
|
|
|||
|
<img src="images/lab06/media/image49.png" style="width:6.12586in;height:5.68829in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. Click “Test connection” and you should see a “Connection successful”
|
|||
|
message. Click Create.
|
|||
|
|
|||
|
<img src="images/lab06/media/image50.png" style="width:6.59467in;height:0.84387in" alt="A picture containing graphical user interface Description automatically generated" />
|
|||
|
|
|||
|
1. Change the Pipeline ID type to “Pipeline endpoint ID”, choose the
|
|||
|
CustomerChurnEP endpoint, choose the pipeline version you want (possibly
|
|||
|
version 0), and set the Experiment name to SynapseAutomatedChurn:
|
|||
|
|
|||
|
<img src="images/lab06/media/image51.png" style="width:7.5in;height:3.54375in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. Click the Debug button at the top of the
|
|||
|
RefreshCustomerChargesAndChurnML pipeline. After 15 minutes or so the
|
|||
|
debug run should complete successfully. (Tip: If you don’t see the
|
|||
|
Output tab, click on the canvas background rather than on a module in
|
|||
|
the pipeline.) Mouse over the Execute Churn ML activity row and click
|
|||
|
the eyeglasses icon to see more details about that execution. Click the
|
|||
|
link to see details from the Azure ML pipeline execution.
|
|||
|
|
|||
|
<img src="images/lab06/media/image52.png" style="width:7.5in;height:3.33472in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. Click Publish all:
|
|||
|
|
|||
|
<img src="images/lab06/media/image53.png" style="width:4.13599in;height:0.30213in" />
|
|||
|
|
|||
|
1. Confirm the two objects which are changed:
|
|||
|
|
|||
|
<img src="images/lab06/media/image54.png" style="width:6.34464in;height:2.80247in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
## Step 2: Schedule the Synapse pipeline
|
|||
|
|
|||
|
1. Click the Add trigger button at the top of the
|
|||
|
RefreshCustomerChargesAndChurnML Synapse pipeline and choose New/Edit:
|
|||
|
|
|||
|
<img src="images/lab06/media/image55.png" style="width:7.5in;height:2.19306in" alt="Graphical user interface, text, application Description automatically generated" />
|
|||
|
|
|||
|
1. On the Add triggers pane, choose +New from the dropdown:
|
|||
|
|
|||
|
<img src="images/lab06/media/image56.png" style="width:6.34464in;height:1.64606in" alt="Graphical user interface, application Description automatically generated" />
|
|||
|
|
|||
|
1. Name the trigger Daily3am. Set the Start date to 3:00AM tomorrow. Choose
|
|||
|
your time zone. Change the recurrence to every 1 day. Then click OK two
|
|||
|
times.
|
|||
|
|
|||
|
<img src="images/lab06/media/image57.png" style="width:6.45923in;height:9.1992in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. Click Publish all and confirm the trigger to be published. Click
|
|||
|
Publish.
|
|||
|
|
|||
|
<img src="images/lab06/media/image58.png" style="width:5.6362in;height:2.35449in" alt="Graphical user interface, text, application, email Description automatically generated" />
|
|||
|
|
|||
|
1. If you ever want to adjust the trigger, go to the Manage left nav,
|
|||
|
Triggers left nav. Mouse over the Daily3am trigger and click the start
|
|||
|
or stop icon to toggle whether this trigger is active. (If there was an
|
|||
|
error activating the trigger during publishing, you can retry
|
|||
|
starting/activating the trigger on this screen.)
|
|||
|
|
|||
|
<img src="images/lab06/media/image59.png" style="width:7.5in;height:3.56042in" alt="Graphical user interface, text, application Description automatically generated" />
|
|||
|
|
|||
|
## Step 3: Schedule the data source refresh in Customer Insights
|
|||
|
|
|||
|
1. At <https://home.ci.ai.dynamics.com/> on the Data… Data sources left
|
|||
|
nav, click the … next to AzureMlResults and Edit refresh settings.
|
|||
|
|
|||
|
<img src="images/lab06/media/image60.png" style="width:7.5in;height:3.71181in" alt="Graphical user interface, text, application Description automatically generated" />
|
|||
|
|
|||
|
1. Choose “Refresh automatically” and set it to refresh on specific days
|
|||
|
and times, choose Daily, and choose a time which is 30 minutes after the
|
|||
|
Synapse pipeline is scheduled to run. Pay attention to the time zone.
|
|||
|
Click Create.
|
|||
|
|
|||
|
<img src="images/lab06/media/image61.png" style="width:7.5in;height:2.79653in" alt="Graphical user interface, text, application Description automatically generated" />
|
|||
|
|
|||
|
# Exercise 3: Cleanup Azure resources (optional)
|
|||
|
|
|||
|
When you are **COMPLETELY DONE** with all your work, you can optionally
|
|||
|
delete these Azure resources. Go to the Overview blade of the
|
|||
|
**customer-insights-workshop-rg** in the Azure portal and click the
|
|||
|
**Delete resource group** button. Your Azure resources and all the data
|
|||
|
in them will be **deleted and unrecoverable**.
|
|||
|
|
|||
|
<img src="images/lab06/media/image62.png" style="width:7.29268in;height:2.3024in" alt="Graphical user interface, application Description automatically generated" />
|
|||
|
|
|||
|
# Summary
|
|||
|
|
|||
|
In this lab, you learned how to automate the refresh and transformation
|
|||
|
of a key customer charges file with Synapse pipelines and Mapping data
|
|||
|
flows. Then you learned how to automate the execution of the Azure ML
|
|||
|
pipeline using Synapse pipelines and how to automate the refresh of data
|
|||
|
into Customer Insights.
|