Updated Deployment Instructions
This commit is contained in:
John Rampono
Родитель
ca31e3d799
Коммит
1c2950b1af
|
|
@ -1 +1,4 @@
|
|||
# Deploy Data Factory Artefacts
|
||||
- In the Azure portal navigate to the Azure Data Factory instance that was created during the ARM template deployment process.
|
||||
- Connect this instance to your cloned
|
||||
- Give MSI access to Azure Function in ADF
|
||||
|
|
@ -5,7 +5,3 @@
|
|||
- Deploy the database project to the new Azure SQL Server instance that has been created in your sandbox environment by the deployment process.
|
||||
- Run the Post Deployment scripts located at \solution\Database\ADSGoFastDatabase\ADSGoFastDatabase\Scripts\. Your newly created database should be the target for these scripts.
|
||||
|
||||
## Configuration
|
||||
- Add AAD Group to SQL Server
|
||||
- Add MSI Access to ADSGoFast for Azure Functions
|
||||
|
||||
|
|
|
|||
|
|
@ -1 +1,75 @@
|
|||
# Deploy and Configure Function Application
|
||||
# Deploy and Configure Function Application
|
||||
|
||||
- Open the Azure Function project located at "\solution\FunctionApp\" in Visual Studio
|
||||
- Deploy the solution to the App Service Environment created by the ARM deployment script.
|
||||
- Once deployed you will need to set the Function App settings via the Azure Portal. Use the **App Settings Template** provided below to create the appropriate settings. Note, you may not have some of the required information yet as some of it is created later in the deployment and configuration process. If you don't know what to fill out just leave a placeholder value there for now.
|
||||
- For local development and testing create a "Local.Settings" file in the solution using the **App Settings Template** template. Note, you may not have some of the required information yet as some of it is created later in the deployment and configuration process. If you don't know what to fill out just leave a placeholder value there for now.
|
||||
|
||||
|
||||
**App Settings Template**
|
||||
```jsonc
|
||||
{
|
||||
"AzureWebJobsStorage": "UseDevelopmentStorage=true", //Only needed for local development environment
|
||||
|
||||
"FUNCTIONS_WORKER_RUNTIME": "dotnet",
|
||||
|
||||
"AZURE_TENANT_ID": "72f988bf-86f1-41af-91ab-2d7cd011db47",
|
||||
|
||||
"TenantId": "72f988bf-86f1-41af-91ab-2d7cd011db47",
|
||||
|
||||
"AZURE_CLIENT_ID": "###", //Service Prinicpal Client Id. Only needed for local development environment (note repeated below due to use of legacy auth provider AND new auth provider)
|
||||
|
||||
"ApplicationId": "####", //Service Prinicpal Client Id. Only needed for local development environment
|
||||
|
||||
"AZURE_CLIENT_SECRET": "####", //Service Prinicpal Auth Key. Only needed for local development environment (note repeated below due to use of legacy auth provider AND new auth provider)
|
||||
|
||||
"AuthenticationKey": "####", //Service Prinicpal Auth Key. Only needed for local development environment
|
||||
|
||||
"UseMSI": true, //Set to true in Azure function deployments and false for local developement deployments
|
||||
|
||||
"FrameworkWideMaxConcurrency": 400, //Max number of concurrent tasks supported
|
||||
|
||||
"AdsGoFastTaskMetaDataDatabaseServer": "#######.database.windows.net", //Address of the framework configuration database
|
||||
|
||||
"AdsGoFastTaskMetaDataDatabaseName": "AdsGoFast",//Database name of the framework configuration database
|
||||
|
||||
"AdsGoFastTaskMetaDataDatabaseUseTrustedConnection": false, //Only needed for local development environment
|
||||
|
||||
"TaskMetaDataStorageAccount": "###", //Only needed for local development environment when we don't want to use a configuration database
|
||||
|
||||
"TaskMetaDataStorageContainer": "###",//Only needed for local development environment when we don't want to use a configuration database
|
||||
|
||||
"TaskMetaDataStorageFolder": "###",//Only needed for local development environment when we don't want to use a configuration database
|
||||
|
||||
"SQLTemplateLocation": ".\\SqlTemplates\\", //Location of SQL Template files ... for local development you should not need to change this. For cloud delployment you need to change to D:\home\site\wwwroot\SqlTemplates\
|
||||
|
||||
"KQLTemplateLocation": ".\\KqlTemplates\\", //Location of KQL Template files ... for local development you should not need to change this. For cloud delployment you need to change to D:\home\site\wwwroot\KqlTemplates\
|
||||
|
||||
"HTMLTemplateLocation": ".\\HTMLEmailTemplates\\", /* !!!!!!! New !!!!!!! */ //Location of Email Template files ... for local development you should not need to change this. For cloud delployment you need to change to D:\home\site\wwwroot\HTMLEmailTemplates\
|
||||
|
||||
"EnablePrepareFrameworkTasks": true, //Set to false to "turn-off" the prepare tasks function for local development environments.
|
||||
|
||||
"EnableRunFrameworkTasks": true, //Set to false to "turn-off" the run tasks function for local development environments.
|
||||
|
||||
"EnableGetADFStats": true, //Set to false to "turn-off" the get ADF stats functions for local development environments.
|
||||
|
||||
"AzureFunctionURL": "http://localhost:7071", //Set to your Azure function App base address for cloud deployments.
|
||||
|
||||
"GetSASUriSendEmailHttpTriggerAzureFunctionKey": "#####", //Set to the value of the Azure Function key. This allows the main functions in the Azure function app to call the GetSASUriSendEmailHttpTrigger function via an HTTP Post.
|
||||
|
||||
"RunFrameworkTasksHttpTriggerAzureFunctionKey": "#####", //Set to the value of the Azure Function key. This allows the main functions in the Azure function app to call the RunFrameworkTasksHttpTriggerr function via an HTTP request.
|
||||
|
||||
"SENDGRID_APIKEY": "##########",//Set to your Sendgrid Key
|
||||
|
||||
"AZStorageCacheFileListHttpTriggerAzureFunctionKey": "NA",/* !!!!!!! New !!!!!!! */ //Set to the value of the Azure Function key. This allows the main functions in the Azure function app to call the AZStorageCacheFileListHttpTrigger function via an HTTP request.
|
||||
|
||||
"DefaultSentFromEmailAddress": "noreply@######.com",/* !!!!!!! New !!!!!!! */ //Set to default sent from address (system wide)
|
||||
|
||||
"DefaultSentFromEmailName": "Ads Go Fast (No Reply)",/* !!!!!!! New !!!!!!! */ //Set to default sent from address name (system wide)
|
||||
|
||||
"GenerateTaskObjectTestFiles": false,/* !!!!!!! New !!!!!!! */ //Provides the ability to generate UnitTest files instead of calling ADF or AF for task processing
|
||||
|
||||
"TaskObjectTestFileLocation": ".\\UnitTestResults\\"/* !!!!!!! New !!!!!!! */ //Location of UnitTest Files... for local development you should not need to change this. For cloud delployment you need to change to D:\home\site\wwwroot\UnitTestResults\
|
||||
|
||||
}
|
||||
```
|
||||
|
|
@ -1 +1,95 @@
|
|||
# Deploy and Configure Web Application
|
||||
# Deploy and Configure Web Application
|
||||
|
||||
* Open the Web Application solution found at "\solution\WebApplication\WebApplication.sln".
|
||||
* Restore Nuget Packages
|
||||
* Restore Libman packages
|
||||
* Publish Application to the App Service Plan created during deployment
|
||||
* Once published you will need to set the Web Application settings using Kudu. Use the **App Settings Template** provided below to create the appropriate settings. Note, you may not have some of the required information yet as some of it is created later in the deployment and configuration process. If you don't know what to fill out just leave a placeholder value there for now.
|
||||
* For local development and testing create an "AppSettings.json" file in the solution using the **App Settings Template** template. Note, you may not have some of the required information yet as some of it is created later in the deployment and configuration process. If you don't know what to fill out just leave a placeholder value there for now.
|
||||
|
||||
**App Settings Template**
|
||||
```jsonc
|
||||
{
|
||||
"Logging": {
|
||||
"LogLevel": {
|
||||
"Default": "Information",
|
||||
"Microsoft": "Warning",
|
||||
"Microsoft.Hosting.Lifetime": "Information"
|
||||
}
|
||||
},
|
||||
"AZURE_TENANT_ID": "" //Azure Tenant Id,
|
||||
"AZURE_CLIENT_ID": "" //Service Prinicpal Client Id. Only needed for local development environment,
|
||||
"AZURE_CLIENT_SECRET": "" //Service Prinicpal secret. Only needed for local development environment,
|
||||
"UseMSI": false ///Set to true for Azure deployments and false for local developement deployments,
|
||||
"AdsGoFastTaskMetaDataDatabaseServer": "" //Address of the framework configuration database,
|
||||
"AdsGoFastTaskMetaDataDatabaseName": "" //Db name of the framework configuration database,
|
||||
"AppInsightsWorkspaceId": "" //Application Insights Workspace Id for the Azure Function Application that you deployed previously. This allows the web applicication to display function activity information for monitoring purposes ,
|
||||
"AdGroups": [
|
||||
],
|
||||
"AllowedHosts": "*",
|
||||
"SecurityModelOptions": {
|
||||
"SecurityRoles": [
|
||||
{
|
||||
"SecurityGroupId": "",
|
||||
"Name": "Administrators",
|
||||
"AllowActions": [
|
||||
]
|
||||
}
|
||||
],
|
||||
"GlobalAllowActions": [
|
||||
"ADFActivityErrors",
|
||||
"ADFPipelineStats",
|
||||
"AFExecutionSummary",
|
||||
"AFLogMonitor",
|
||||
"Dashboard",
|
||||
"DataFactory",
|
||||
"FrameworkTaskRunner",
|
||||
"FrameworkTaskRunnerDapper",
|
||||
"ReportsAndStatistics",
|
||||
"ScheduleInstance",
|
||||
"ScheduleMaster",
|
||||
"SourceAndTargetSystems",
|
||||
"SourceAndTargetSystemsJsonSchema",
|
||||
"SubjectArea",
|
||||
"TaskGroup",
|
||||
"TaskGroupDependency",
|
||||
"TaskInstance",
|
||||
"TaskInstanceExecution",
|
||||
"TaskMaster",
|
||||
"TaskMasterWaterMark",
|
||||
"TaskType",
|
||||
"TaskTypeMapping",
|
||||
"Wizards"
|
||||
|
||||
],
|
||||
"GlobalDenyActions": [
|
||||
"Customisations.Delete",
|
||||
"DataFactory.Delete",
|
||||
"FrameworkTaskRunner.Delete",
|
||||
"FrameworkTaskRunnerDapper.Delete",
|
||||
"ScheduleInstance.Delete",
|
||||
"ScheduleMaster.Delete",
|
||||
"SourceAndTargetSystems.Delete",
|
||||
"SourceAndTargetSystemsJsonSchema.Delete",
|
||||
"TaskGroup.Delete",
|
||||
"TaskGroupDependency.Delete",
|
||||
"TaskGroupDependency.DeletePlus",
|
||||
"TaskInstance.Delete",
|
||||
"TaskMaster.Delete",
|
||||
"TaskMasterWaterMark.Delete",
|
||||
"TaskType.Delete",
|
||||
"TaskTypeMapping.Delete"
|
||||
]
|
||||
},
|
||||
//Details below are specificly for AAD integration. You will need to create an App Registration specifically for AAD integration. Do this using the Azure Portal and enable AAD integration for your web app either using the express settup method or by doing it manually. Fill in the relevant info below
|
||||
"AzureAd": {
|
||||
"Instance": "https://login.microsoftonline.com/",
|
||||
"Domain": "",
|
||||
"TenantId": "",
|
||||
"ClientId": "",
|
||||
"CallbackPath": "/signin-oidc",
|
||||
"SignedOutCallbackPath ": "/signout-callback-oidc"
|
||||
}
|
||||
|
||||
}
|
||||
```
|
||||
|
|
@ -60,7 +60,10 @@ In this section you will use automated deployment and ARM templates to automate
|
|||
|
||||
|
||||
|
||||
# Solution Deployment and Configuration
|
||||
## Obtain the source code
|
||||
Get a copy of the source code by either donwloading it as a zip file from git or cloning the repository.
|
||||
|
||||
## Solution Deployment and Configuration
|
||||
In this section you will use Azure DevOps to deploy the Database project, Data Factory components (Pipeline, Datasets, LinkedServices and Integration Runtime), Azure Function Application, and Administrative Web Application. Click through the links below and follow the instructions provided.
|
||||
|
||||
|
||||
|
|
@ -74,13 +77,33 @@ In this section you will use Azure DevOps to deploy the Database project, Data F
|
|||
|
||||
|
||||
## Access provisioning
|
||||
The following sections will guide you through the processes required to set-up the required rights and privleges for service accounts and credentials used.
|
||||
|
||||
- MSI
|
||||
- Azure Data Factory uses MSI to authenticate to staging database (Azure SQL Database), Azure KeyVault and Azure Storage;
|
||||
- Azure Data Factory use SQL Authentication to authenticate to On-Prem SQL Server;
|
||||
- Azure Functions uses MSI to authenticate to ADSGoFast database (Azure SQL Database), Azure Storage, Azure Data Factory;
|
||||
### Deployment Credentials
|
||||
|
||||
# ADD to ARM Template
|
||||
- Azure Data Factory MSI - Grant the managed service identity for Azure Data Factory rights as per the list below:
|
||||
- Adventureworks Azure SQL Sample database [Read]
|
||||
- Key Vault [Read Secrets]
|
||||
- Datalake Storage Accounts [Blob Storage Contributor Role]
|
||||
- On-Prem SQL Server (Proxied by Azure VM in Demo Environment) [Read];
|
||||
- Azure Functions MSI - Grant the managed service identity rights as per the list below
|
||||
- ADS Go Fast metadata database [Read,Write and Execute]
|
||||
- Datalake Storage Accounts [Blob Storage Contributor Role]
|
||||
- Azure Data Factory [Contributor Role];
|
||||
- Application Insights for Function App & Web App [Read];
|
||||
- Log Analytics for Data Factory [Read]
|
||||
- Web Application MSI - Grant the managed service identity rights as per the list below
|
||||
- ADS Go Fast metadata database [Read,Write and Execute]
|
||||
- Datalake Storage Accounts [Blob Storage Contributor Role]
|
||||
- Application Insights for Function App & Web App [Read];
|
||||
- Log Analytics for Data Factory [Read]
|
||||
|
||||
### Local Development Credentials
|
||||
|
||||
- To facilitate local development Service Principals are used. Provision a service principal to act as a proxy for each of the MSI's in the list above. Once provisioned, grant these MSI's the equivalent rights so that they can appropriatley mimic the MSI's. Note, it is recommended that you only provision these service principals in NON-PRODUCTION environments.
|
||||
|
||||
# TODO
|
||||
## ADD to ARM Template
|
||||
- Create Staging database
|
||||
- Create Storage Container "datalakeraw and datalakelanding" on Data Lake Storage
|
||||
- Create Transient Storage Account and add Containers "transientin and transientout"
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче