Scenario Tester Deployment Scripts (#161)
* Initial commit for scenario tester arm deployment * Initial commit for deployment of Scenario Tester through an ARM Template * Some improvements: Event hub Id detection based in if it is a iot device or not * Add smart is data bricks function * Error fix: connection string property was not correctly set * Initial commit for deployment scripts and utilities * Deploy a default parameter file * Configure properly a system identity to the new scenario tester app + grant access to the service key vault. Do the same for spark keyvault as well. Make the app always on too. * Retrieve most of the information in the deployment resources existing in the environment already * Fix some errors in the script and template. Load from package was not really supported and there was a typo in the template. * typo fix in blob storage uri * Use the correct connection string from the array * Adding missing entries into the spark kv + make cert validation an argument to the deployment script entrypoint * Pass key vault names as parameters to json deployment file + make data bricks token resource conditional to the environment spark type * Add comments for deployment scripts for scenario tester. * Add a few more details * Add basic readme file for scenario tester directory * Enrich readme even further
This commit is contained in:
jozavala
GitHub
Родитель
ccfccdc1eb
Коммит
86e959af67
|
|
@ -0,0 +1,27 @@
|
|||
# Scenario Tester
|
||||
|
||||
Scenario Tester is a tool that proactively performs http calls to data-accelerator backend and asserts that the expected output is returned by running a set of user-defined scenario runs. Each scenario is run periodically and results are tracked and stored in a database.
|
||||
|
||||
# Install
|
||||
|
||||
## Prerequisites
|
||||
From cloud installation steps [link](https://github.com/Microsoft/data-accelerator/wiki/Cloud-deployment) retrieve the used inputs from common.parameters.txt and admin.parameters.txt and from deployed data-accelerator resources. Specifically we require the following:
|
||||
|
||||
- SubscriptionId: Subscription guid where data-accelerator was previously deployed
|
||||
- TenantId: Tenant where the subscription belongs to
|
||||
- ResourceGroup: Resource group where data-accelerator was previously deployed
|
||||
- Application id and secret: Service principal id and secret used in the admin steps
|
||||
- Keyvault base name prefix: Prefix chose by the data-accelerator deployment script. You can look for the prefix manually by looking at the data-accelerator current keyvault secret names
|
||||
|
||||
## Steps
|
||||
Execute deployResources.ps1, for example:
|
||||
|
||||
```
|
||||
deployResources.ps1 -tenantId $tenantId -subscriptionId $subscriptionId -resourceGroupName $rg -applicationId $appId -appSecretKey $appSecret -kvBaseNamePrefix $kvPrefix
|
||||
```
|
||||
|
||||
# Development
|
||||
|
||||
Add your scenario creating a new class inheriting IJob and place it at [Jobs](https://github.com/microsoft/data-accelerator/tree/stable/Services/JobRunner/Jobs) directory. Then register your scenario in [JobRunner class](https://github.com/microsoft/data-accelerator/blob/master/Services/JobRunner/JobRunner.cs#L274).
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,90 @@
|
|||
{
|
||||
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
|
||||
"contentVersion": "1.0.0.0",
|
||||
"parameters": {
|
||||
"tenantId": {
|
||||
"value" : "$tenantId"
|
||||
},
|
||||
"productName": {
|
||||
"value" : "$productName"
|
||||
},
|
||||
"applicationObjectId": {
|
||||
"value" : "$applicationObjectId"
|
||||
},
|
||||
"applicationId": {
|
||||
"value" : "$applicationId"
|
||||
},
|
||||
"applicationIdentifierUri": {
|
||||
"value" : "$applicationIdentifierUri"
|
||||
},
|
||||
"authorityUri": {
|
||||
"value" : "$authorityUri"
|
||||
},
|
||||
"serviceUrl": {
|
||||
"value" : "$serviceUrl"
|
||||
},
|
||||
"secretKey": {
|
||||
"value" : "$secretKey"
|
||||
},
|
||||
"jobRunnerName": {
|
||||
"value" : "$jobRunnerName"
|
||||
},
|
||||
"sparkType": {
|
||||
"value" : "$sparkType"
|
||||
},
|
||||
"databricksToken": {
|
||||
"value" : "$databricksToken"
|
||||
},
|
||||
"eventHubConnectionString": {
|
||||
"value" : "$eventHubConnectionString"
|
||||
},
|
||||
"eventHubName": {
|
||||
"value" : "$eventHubName"
|
||||
},
|
||||
"isIotHub": {
|
||||
"value" : "$isIotHub"
|
||||
},
|
||||
"blobUri": {
|
||||
"value" : "$blobUri"
|
||||
},
|
||||
"sparkStorageAccountName": {
|
||||
"value" : "$sparkStorageAccountName"
|
||||
},
|
||||
"flowName": {
|
||||
"value" : "$flowName"
|
||||
},
|
||||
"normalizationSnippet": {
|
||||
"value" : "$normalizationSnippet"
|
||||
},
|
||||
"referenceDataUri": {
|
||||
"value" : "$referenceDataUri"
|
||||
},
|
||||
"udfSampleUri": {
|
||||
"value" : "$udfSampleUri"
|
||||
},
|
||||
"kvServicesName": {
|
||||
"value" : "$kvServicesName"
|
||||
},
|
||||
"kvSparkName": {
|
||||
"value" : "$kvSparkName"
|
||||
},
|
||||
"skipServerCertificateValidation": {
|
||||
"value" : "$skipServerCertificateValidation"
|
||||
},
|
||||
"primaryQueueName": {
|
||||
"value" : "$primaryQueueName"
|
||||
},
|
||||
"activeQueueName": {
|
||||
"value" : "$activeQueueName"
|
||||
},
|
||||
"testQueueName": {
|
||||
"value" : "$testQueueName"
|
||||
},
|
||||
"seconds": {
|
||||
"value" : "$seconds"
|
||||
},
|
||||
"aspNetCoreDetailedErrorsEnabled": {
|
||||
"value" : "$aspNetCoreDetailedErrorsEnabled"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,604 @@
|
|||
{
|
||||
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
|
||||
"contentVersion": "1.0.0.0",
|
||||
"parameters": {
|
||||
"tenantId": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Provide the tenant id"
|
||||
}
|
||||
},
|
||||
"productName": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Product name chosen for this environment"
|
||||
}
|
||||
},
|
||||
"applicationObjectId": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "The principal object id of scenario tester application"
|
||||
}
|
||||
},
|
||||
"applicationId": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Provide the application id for scenario tester"
|
||||
}
|
||||
},
|
||||
"applicationIdentifierUri": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Application identifier uri for scenario tester"
|
||||
}
|
||||
},
|
||||
"authorityUri": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Authority uri"
|
||||
}
|
||||
},
|
||||
"serviceUrl": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Url to the scenario tester target service"
|
||||
}
|
||||
},
|
||||
"secretKey": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Provide the secret key for scenario tester application"
|
||||
}
|
||||
},
|
||||
"jobRunnerName": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Scenario tester webapp name in environment"
|
||||
}
|
||||
},
|
||||
"sparkType": {
|
||||
"defaultValue": "hdinsight",
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Use either hdinsight or databricks"
|
||||
}
|
||||
},
|
||||
"databricksToken": {
|
||||
"defaultValue": "",
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Data bricks token for databricks configuration"
|
||||
}
|
||||
},
|
||||
"eventHubConnectionString": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "The event hub connection string used by scenario tester"
|
||||
}
|
||||
},
|
||||
"eventHubName": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Event hub name used for scenarios"
|
||||
}
|
||||
},
|
||||
"isIotHub": {
|
||||
"defaultValue": "true",
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Tells whether the event hub is a iot hub"
|
||||
}
|
||||
},
|
||||
"blobUri": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Blob uri to the scenario tester flow payload"
|
||||
}
|
||||
},
|
||||
"sparkStorageAccountName": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Name of the storage account name"
|
||||
}
|
||||
},
|
||||
"flowName": {
|
||||
"defaultValue": "scenariotest",
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Flow name used for scenarios"
|
||||
}
|
||||
},
|
||||
"normalizationSnippet": {
|
||||
"defaultValue": "SystemProperties AS _SystemProperties\\r\\nProperties AS _Properties\\r\\nstringToTimestamp(Raw.deviceDetails.eventTime) AS eventTimeStamp\\r\\nRaw.*",
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Normalization snippet used for scenarios"
|
||||
}
|
||||
},
|
||||
"referenceDataUri": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Uri to reference data for scenario tester"
|
||||
}
|
||||
},
|
||||
"udfSampleUri": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Uri to udf sample for scenario tester"
|
||||
}
|
||||
},
|
||||
"kvServicesName": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Name of the key vault for services"
|
||||
}
|
||||
},
|
||||
"kvSparkName": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"description": "Name of the key vault for spark"
|
||||
}
|
||||
},
|
||||
"skipServerCertificateValidation": {
|
||||
"defaultValue": "false",
|
||||
"type": "string"
|
||||
},
|
||||
"primaryQueueName": {
|
||||
"defaultValue": "job_runner",
|
||||
"type": "string"
|
||||
},
|
||||
"activeQueueName": {
|
||||
"defaultValue": "job_runner",
|
||||
"type": "string"
|
||||
},
|
||||
"testQueueName": {
|
||||
"defaultValue": "job_runner",
|
||||
"type": "string"
|
||||
},
|
||||
"seconds": {
|
||||
"defaultValue": "10",
|
||||
"type": "string"
|
||||
},
|
||||
"aspNetCoreDetailedErrorsEnabled": {
|
||||
"defaultValue": "false",
|
||||
"type": "string"
|
||||
},
|
||||
"secretsPermissions": {
|
||||
"type": "array",
|
||||
"defaultValue": [
|
||||
"get",
|
||||
"list",
|
||||
"set"
|
||||
],
|
||||
"metadata": {
|
||||
"description": "Specifies the permissions to secrets in the vault. Valid values are: all, get, list, set, delete, backup, restore, recover, and purge."
|
||||
}
|
||||
},
|
||||
"eventHubConnectionStringKVName": {
|
||||
"type": "string"
|
||||
},
|
||||
"referenceDataKVName": {
|
||||
"type": "string"
|
||||
},
|
||||
"udfSampleKVName": {
|
||||
"type": "string"
|
||||
},
|
||||
"databricksTokenKVName": {
|
||||
"type": "string"
|
||||
},
|
||||
"clientIdKVName": {
|
||||
"type": "string"
|
||||
},
|
||||
"secretKeyKVName": {
|
||||
"type": "string"
|
||||
},
|
||||
"testClientId": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"variables": {
|
||||
"jobRunnerPlanName": "[concat(parameters('jobRunnerName'), 'Plan')]",
|
||||
"kvUriPrefix": "[if(DataX.isDatabricksType(parameters('sparkType')), 'secretscope', 'keyvault')]",
|
||||
"kvServicesBaseUri": "[concat(variables('kvUriPrefix'), '://', parameters('kvServicesName'))]",
|
||||
"kvSparkBaseUri": "[concat(variables('kvUriPrefix'), '://', parameters('kvSparkName'))]",
|
||||
"kvVaultId": "[resourceId('Microsoft.KeyVault/vaults', parameters('kvServicesName'))]",
|
||||
"sbNamespace": "[concat(parameters('productName'), 'table')]",
|
||||
"cosmosDBName": "[concat(parameters('productName'), '-cosmostable')]",
|
||||
"authRulesName": "RootManageSharedAccessKey",
|
||||
"authRulesNameFull": "[concat(variables('sbNamespace'), '/', variables('authRulesName'))]"
|
||||
},
|
||||
"functions": [
|
||||
{
|
||||
"namespace": "DataX",
|
||||
"members": {
|
||||
"isDatabricksType": {
|
||||
"parameters": [
|
||||
{
|
||||
"name": "type",
|
||||
"type": "string"
|
||||
}
|
||||
],
|
||||
"output": {
|
||||
"type": "bool",
|
||||
"value": "[bool(if(equals(parameters('type'), 'databricks'), 'true', 'false'))]"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"resources": [
|
||||
{
|
||||
"type": "Microsoft.KeyVault/vaults/secrets",
|
||||
"name": "[concat(parameters('kvServicesName'), '/', parameters('testClientId'))]",
|
||||
"comments": "Api client id that will be whitelisted in gateway",
|
||||
"apiVersion": "2015-06-01",
|
||||
"location": "[resourceGroup().location]",
|
||||
"properties": {
|
||||
"value": "[concat(parameters('applicationObjectId'), '.', parameters('tenantId'))]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.KeyVault/vaults/secrets",
|
||||
"name": "[concat(parameters('kvServicesName'), '/', parameters('clientIdKVName'))]",
|
||||
"comments": "Api client id that scenario tester uses to make api calls",
|
||||
"apiVersion": "2015-06-01",
|
||||
"location": "[resourceGroup().location]",
|
||||
"properties": {
|
||||
"value": "[parameters('applicationId')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.KeyVault/vaults/secrets",
|
||||
"name": "[concat(parameters('kvServicesName'), '/', parameters('secretKeyKVName'))]",
|
||||
"comments": "Api secret key that scenario tester uses to make api calls",
|
||||
"apiVersion": "2015-06-01",
|
||||
"location": "[resourceGroup().location]",
|
||||
"properties": {
|
||||
"value": "[parameters('secretKey')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"condition": "[DataX.isDatabricksType(parameters('sparkType'))]",
|
||||
"type": "Microsoft.KeyVault/vaults/secrets",
|
||||
"name": "[concat(parameters('kvServicesName'), '/', parameters('databricksTokenKVName'))]",
|
||||
"comments": "Add databricks token to the keyvault",
|
||||
"apiVersion": "2015-06-01",
|
||||
"location": "[resourceGroup().location]",
|
||||
"properties": {
|
||||
"value": "[parameters('databricksToken')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.KeyVault/vaults/secrets",
|
||||
"name": "[concat(parameters('kvServicesName'), '/', parameters('eventHubConnectionStringKVName'))]",
|
||||
"comments": "Connection string to event hub used for scenarios",
|
||||
"apiVersion": "2015-06-01",
|
||||
"location": "[resourceGroup().location]",
|
||||
"properties": {
|
||||
"value": "[parameters('eventHubConnectionString')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.KeyVault/vaults/secrets",
|
||||
"name": "[concat(parameters('kvServicesName'), '/', parameters('udfSampleKVName'))]",
|
||||
"comments": "Connection uri to udf sample",
|
||||
"apiVersion": "2015-06-01",
|
||||
"location": "[resourceGroup().location]",
|
||||
"properties": {
|
||||
"value": "[parameters('udfSampleUri')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.KeyVault/vaults/secrets",
|
||||
"name": "[concat(parameters('kvServicesName'), '/', parameters('referenceDataKVName'))]",
|
||||
"comments": "Connection uri to devices reference data",
|
||||
"apiVersion": "2015-06-01",
|
||||
"location": "[resourceGroup().location]",
|
||||
"properties": {
|
||||
"value": "[parameters('referenceDataUri')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"condition": "[DataX.isDatabricksType(parameters('sparkType'))]",
|
||||
"type": "Microsoft.KeyVault/vaults/secrets",
|
||||
"name": "[concat(parameters('kvSparkName'), '/', parameters('databricksTokenKVName'))]",
|
||||
"comments": "Add databricks token to the keyvault",
|
||||
"apiVersion": "2015-06-01",
|
||||
"location": "[resourceGroup().location]",
|
||||
"properties": {
|
||||
"value": "[parameters('databricksToken')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.KeyVault/vaults/secrets",
|
||||
"name": "[concat(parameters('kvSparkName'), '/', parameters('eventHubConnectionStringKVName'))]",
|
||||
"comments": "Connection string to event hub used for scenarios",
|
||||
"apiVersion": "2015-06-01",
|
||||
"location": "[resourceGroup().location]",
|
||||
"properties": {
|
||||
"value": "[parameters('eventHubConnectionString')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.KeyVault/vaults/secrets",
|
||||
"name": "[concat(parameters('kvSparkName'), '/', parameters('udfSampleKVName'))]",
|
||||
"comments": "Connection uri to udf sample",
|
||||
"apiVersion": "2015-06-01",
|
||||
"location": "[resourceGroup().location]",
|
||||
"properties": {
|
||||
"value": "[parameters('udfSampleUri')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.KeyVault/vaults/secrets",
|
||||
"name": "[concat(parameters('kvSparkName'), '/', parameters('referenceDataKVName'))]",
|
||||
"comments": "Connection uri to devices reference data",
|
||||
"apiVersion": "2015-06-01",
|
||||
"location": "[resourceGroup().location]",
|
||||
"properties": {
|
||||
"value": "[parameters('referenceDataUri')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.ServiceBus/namespaces",
|
||||
"apiVersion": "2017-04-01",
|
||||
"name": "[variables('sbNamespace')]",
|
||||
"comments": "Service bus for scenario tester job queue",
|
||||
"location": "[resourceGroup().location]",
|
||||
"sku": {
|
||||
"name": "Standard",
|
||||
"tier": "Standard"
|
||||
},
|
||||
"properties": {
|
||||
"provisioningState": "Succeeded",
|
||||
"metricId": "[concat('27de6be1-d524-44f2-9a95-10ab33646cc8:', variables('sbNamespace'))]",
|
||||
"createdAt": null,
|
||||
"updatedAt": null,
|
||||
"serviceBusEndpoint": "[concat('https://', variables('sbNamespace'), '.servicebus.windows.net:443/')]",
|
||||
"status": "Active"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.ServiceBus/namespaces/AuthorizationRules",
|
||||
"apiVersion": "2017-04-01",
|
||||
"name": "[variables('authRulesNameFull')]",
|
||||
"comments": "Add auth rules for service bus",
|
||||
"location": "[resourceGroup().location]",
|
||||
"dependsOn": [
|
||||
"[resourceId('Microsoft.ServiceBus/namespaces', variables('sbNamespace'))]"
|
||||
],
|
||||
"properties": {
|
||||
"rights": [
|
||||
"Listen",
|
||||
"Manage",
|
||||
"Send"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.KeyVault/vaults/secrets",
|
||||
"name": "[concat(parameters('kvServicesName'), '/', 'configgen-sbconn')]",
|
||||
"comments": "Add service bus connection string to key vault",
|
||||
"apiVersion": "2015-06-01",
|
||||
"location": "[resourceGroup().location]",
|
||||
"dependsOn": [
|
||||
"[resourceId('Microsoft.ServiceBus/namespaces/authorizationRules', variables('sbNamespace'), variables('authRulesName'))]"
|
||||
],
|
||||
"properties": {
|
||||
"value": "[listKeys(resourceId(concat('Microsoft.ServiceBus/namespaces/AuthorizationRules'), variables('sbNamespace'), variables('authRulesName')), '2015-08-01').primaryConnectionString]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.ServiceBus/namespaces/queues",
|
||||
"apiVersion": "2017-04-01",
|
||||
"name": "[concat(variables('sbNamespace'), '/job_runner')]",
|
||||
"comments": "Add scenario tester job queue",
|
||||
"location": "[resourceGroup().location]",
|
||||
"dependsOn": [
|
||||
"[resourceId('Microsoft.ServiceBus/namespaces', variables('sbNamespace'))]"
|
||||
],
|
||||
"properties": {
|
||||
"lockDuration": "PT5M",
|
||||
"maxSizeInMegabytes": 5120,
|
||||
"requiresDuplicateDetection": true,
|
||||
"requiresSession": false,
|
||||
"defaultMessageTimeToLive": "P14D",
|
||||
"deadLetteringOnMessageExpiration": true,
|
||||
"enableBatchedOperations": true,
|
||||
"duplicateDetectionHistoryTimeWindow": "PT10M",
|
||||
"maxDeliveryCount": 10,
|
||||
"status": "Active",
|
||||
"autoDeleteOnIdle": "P10675199DT2H48M5.4775807S",
|
||||
"enablePartitioning": true,
|
||||
"enableExpress": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.DocumentDB/databaseAccounts",
|
||||
"comments": "Add cosmos db for scenario tester",
|
||||
"apiVersion": "2015-04-08",
|
||||
"name": "[variables('cosmosDBName')]",
|
||||
"location": "[resourceGroup().location]",
|
||||
"tags": {
|
||||
"defaultExperience": "Azure Table"
|
||||
},
|
||||
"kind": "GlobalDocumentDB",
|
||||
"properties": {
|
||||
"enableAutomaticFailover": false,
|
||||
"enableMultipleWriteLocations": true,
|
||||
"isVirtualNetworkFilterEnabled": false,
|
||||
"virtualNetworkRules": [],
|
||||
"databaseAccountOfferType": "Standard",
|
||||
"consistencyPolicy": {
|
||||
"defaultConsistencyLevel": "BoundedStaleness",
|
||||
"maxIntervalInSeconds": 86400,
|
||||
"maxStalenessPrefix": 1000000
|
||||
},
|
||||
"locations": [
|
||||
{
|
||||
"locationName": "[resourceGroup().location]",
|
||||
"provisioningState": "Succeeded",
|
||||
"failoverPriority": 0
|
||||
}
|
||||
],
|
||||
"capabilities": [
|
||||
{
|
||||
"name": "EnableTable"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.KeyVault/vaults/secrets",
|
||||
"name": "[concat(parameters('kvServicesName'), '/', 'configgen-cosmosdbtableconn')]",
|
||||
"comments": "Add cosmos db connection string to key vault",
|
||||
"apiVersion": "2015-06-01",
|
||||
"location": "[resourceGroup().location]",
|
||||
"dependsOn": [
|
||||
"[resourceId('Microsoft.DocumentDB/databaseAccounts', variables('cosmosDBName'))]"
|
||||
],
|
||||
"properties": {
|
||||
"value": "[listConnectionStrings(resourceId('Microsoft.DocumentDB/databaseAccounts', variables('cosmosDBName')),'2015-04-08').connectionStrings[4].connectionString]"
|
||||
}
|
||||
},
|
||||
|
||||
{
|
||||
"type": "Microsoft.Web/serverfarms",
|
||||
"apiVersion": "2016-09-01",
|
||||
"name": "[variables('jobRunnerPlanName')]",
|
||||
"comments": "Add service plan for job runner app",
|
||||
"location": "[resourceGroup().location]",
|
||||
"sku": {
|
||||
"name": "S2",
|
||||
"tier": "Standard",
|
||||
"size": "S2",
|
||||
"family": "S",
|
||||
"capacity": 2
|
||||
},
|
||||
"kind": "app",
|
||||
"properties": {
|
||||
"name": "[variables('jobRunnerPlanName')]",
|
||||
"perSiteScaling": false,
|
||||
"reserved": false,
|
||||
"targetWorkerCount": 0,
|
||||
"targetWorkerSizeId": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.Web/sites",
|
||||
"apiVersion": "2018-11-01",
|
||||
"name": "[parameters('jobRunnerName')]",
|
||||
"comments": "Deploy Scenario tester",
|
||||
"location": "[resourceGroup().location]",
|
||||
"dependsOn": [
|
||||
"[resourceId('Microsoft.Web/serverfarms', variables('jobRunnerPlanName'))]"
|
||||
],
|
||||
"kind": "app",
|
||||
"properties": {
|
||||
"enabled": true,
|
||||
"hostNameSslStates": [
|
||||
{
|
||||
"name": "[concat(parameters('jobRunnerName'), '.azurewebsites.net')]",
|
||||
"sslState": "Disabled",
|
||||
"hostType": "Standard"
|
||||
},
|
||||
{
|
||||
"name": "[concat(parameters('jobRunnerName'), '.scm.azurewebsites.net')]",
|
||||
"sslState": "Disabled",
|
||||
"hostType": "Repository"
|
||||
}
|
||||
],
|
||||
"serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('jobRunnerPlanName'))]",
|
||||
"reserved": false,
|
||||
"scmSiteAlsoStopped": false,
|
||||
"clientAffinityEnabled": true,
|
||||
"clientCertEnabled": false,
|
||||
"hostNamesDisabled": false,
|
||||
"containerSize": 0,
|
||||
"dailyMemoryTimeQuota": 0,
|
||||
"httpsOnly": true,
|
||||
"siteConfig": {
|
||||
"alwaysOn": true
|
||||
},
|
||||
"appSettings": []
|
||||
},
|
||||
"resources": [
|
||||
{
|
||||
"name": "appsettings",
|
||||
"type": "config",
|
||||
"apiVersion": "2015-08-01",
|
||||
"dependsOn": [
|
||||
"[resourceId('Microsoft.Web/sites', parameters('jobRunnerName'))]"
|
||||
],
|
||||
"properties": {
|
||||
"ASPNETCORE_DETAILEDERRORS": "[parameters('aspNetCoreDetailedErrorsEnabled')]",
|
||||
"DATAX_JobRunner:ServiceUrl": "[parameters('serviceUrl')]",
|
||||
"DATAX_JobRunner:SparkType": "[parameters('sparkType')]",
|
||||
"DATAX_JobRunner:ServiceKeyVaultName": "[parameters('kvServicesName')]",
|
||||
"DATAX_JobRunner:SkipServerCertificateValidation": "[parameters('skipServerCertificateValidation')]",
|
||||
"DATAX_JobRunner:ApplicationIdentifierUri": "[parameters('applicationIdentifierUri')]",
|
||||
"DATAX_JobRunner:MicrosoftAuthority": "[parameters('authorityUri')]",
|
||||
"DATAX_JobRunner:ApplicationId": "[concat(variables('kvServicesBaseUri'), '/configgen-scenarioTester-clientId')]",
|
||||
"DATAX_JobRunner:SecretKey": "[concat(variables('kvServicesBaseUri'), '/configgen-scenarioTester-secretKey')]",
|
||||
"DATAX_JobRunner:DatabricksToken": "[concat(variables('kvServicesBaseUri'), '/scenariotest-info-databricksToken')]",
|
||||
"DATAX_JobRunner:StorageConnection": "[concat(variables('kvServicesBaseUri'), '/configgen-cosmosdbtableconn')]",
|
||||
"DATAX_JobRunner:ServiceBusConnectionString": "[concat(variables('kvServicesBaseUri'), '/configgen-sbconn')]",
|
||||
"DATAX_JobRunner:BlobConnectionString": "[concat(variables('kvServicesBaseUri'), '/configgen-', parameters('sparkStorageAccountName'), '-blobconnectionstring')]",
|
||||
"DATAX_JobRunner:AppInsightsIntrumentationKey": "configgen-aiInstrumentationKey",
|
||||
"DATAX_JobRunner:ActiveQueueName": "[parameters('activeQueueName')]",
|
||||
"DATAX_JobRunner:PrimaryQueueName": "[parameters('primaryQueueName')]",
|
||||
"DATAX_JobRunner:TestQueueName": "[parameters('testQueueName')]",
|
||||
"DATAX_JobRunner:EvenHubConnectionString": "[concat(variables('kvSparkBaseUri'), '/scenariotest-input-eventhubconnectionstring')]",
|
||||
"DATAX_JobRunner:NormalizationSnippet": "[parameters('normalizationSnippet')]",
|
||||
"DATAX_JobRunner:BlobUri": "[parameters('blobUri')]",
|
||||
"DATAX_JobRunner:FlowName": "[parameters('flowName')]",
|
||||
"DATAX_JobRunner:IsIotHub": "[parameters('isIotHub')]",
|
||||
"DATAX_JobRunner:EventHubName": "[parameters('eventHubName')]",
|
||||
"DATAX_JobRunner:Seconds": "[parameters('seconds')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.KeyVault/vaults/accessPolicies",
|
||||
"name": "[concat(parameters('kvServicesName'), '/add')]",
|
||||
"apiVersion": "2018-02-14",
|
||||
"dependsOn": [
|
||||
"[resourceId('Microsoft.Web/sites', parameters('jobRunnerName'))]"
|
||||
],
|
||||
"properties": {
|
||||
"accessPolicies": [
|
||||
{
|
||||
"tenantId": "[parameters('tenantId')]",
|
||||
"objectId": "[reference(resourceId('Microsoft.Web/sites', parameters('jobRunnerName')), '2018-11-01', 'Full').identity.principalId]",
|
||||
"permissions": {
|
||||
"secrets": "[parameters('secretsPermissions')]"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.KeyVault/vaults/accessPolicies",
|
||||
"name": "[concat(parameters('kvSparkName'), '/add')]",
|
||||
"apiVersion": "2018-02-14",
|
||||
"dependsOn": [
|
||||
"[resourceId('Microsoft.Web/sites', parameters('jobRunnerName'))]"
|
||||
],
|
||||
"properties": {
|
||||
"accessPolicies": [
|
||||
{
|
||||
"tenantId": "[parameters('tenantId')]",
|
||||
"objectId": "[reference(resourceId('Microsoft.Web/sites', parameters('jobRunnerName')), '2018-11-01', 'Full').identity.principalId]",
|
||||
"permissions": {
|
||||
"secrets": "[parameters('secretsPermissions')]"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"identity": {
|
||||
"type": "SystemAssigned"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
Различия файлов скрыты, потому что одна или несколько строк слишком длинны
Различия файлов скрыты, потому что одна или несколько строк слишком длинны
|
|
@ -0,0 +1,108 @@
|
|||
#
|
||||
# DeployResources Deploy Script
|
||||
#
|
||||
# This script deploys all the resources to an azure subscription and resource group that are required for the scenario tester job runner.
|
||||
# This asumes that the target subscription contains a data-accelerator environment already deployed and a keyvault already initialized.
|
||||
# This script takes the values from the data-accelerator environment keyvault to initialize the new resources used by scenario tester.
|
||||
# This script also requires that an application id and secret are already created from a registered app, and that registered app is already onboarded
|
||||
# to the Gateway Service. Finally, you can skip certificate validation in case it is required for your test client run cases.
|
||||
#
|
||||
# Output: Scenario tester is deployed into an existing data-accelerator environment
|
||||
#
|
||||
param(
|
||||
# Tenant id where data-accelerator environment is deployed
|
||||
[string]
|
||||
$tenantId,
|
||||
# Subscription id where data-accelerator environment is deployed
|
||||
[string]
|
||||
$subscriptionId,
|
||||
# Resource Group name where data-accelerator environment is deployed
|
||||
[string]
|
||||
$resourceGroupName,
|
||||
# Application id that job runner is going to run in behalf of
|
||||
[string]
|
||||
$applicationId,
|
||||
# Application secret for the target application id
|
||||
[string]
|
||||
$appSecretKey,
|
||||
# Prefix chosen for deployed secrets of data-accelerator main keyvault
|
||||
[string]
|
||||
$kvBaseNamePrefix='configgen',
|
||||
# Name for scenario test flow
|
||||
[string]
|
||||
$flowName='scenariotest',
|
||||
# Skip server certificate validation (TLS check from client side)
|
||||
[string]
|
||||
$skipServerCertificateValidation='false'
|
||||
)
|
||||
Import-Module ./utilities.psm1
|
||||
|
||||
$ErrorActionPreference = "stop"
|
||||
|
||||
Push-Location $PSScriptRoot
|
||||
|
||||
$scenarioTestKVBaseName = $flowName
|
||||
|
||||
$jobRunnerBasePath = $PSScriptRoot
|
||||
$resourcesPath = $jobRunnerBasePath + "\Resources"
|
||||
$templatesPath = $resourcesPath + "\Templates"
|
||||
$parametersPath = $resourcesPath + "\Parameters"
|
||||
$jobRunnerTemplateFile = $templatesPath + "\jobRunner-Template.json"
|
||||
$jobRunnerParameterFile = $parametersPath + "\JobRunner-Parameter.json"
|
||||
|
||||
Write-Host "Template json path: $jobRunnerTemplateFile"
|
||||
|
||||
Write-Host -ForegroundColor Green "Total estimated time to complete: 10 minutes"
|
||||
|
||||
$login = Login -subscriptionId $subscriptionId -tenantId $tenantId
|
||||
$tenantName = $login.tenantName
|
||||
|
||||
$appAccInfo = Get-AppInfo -applicationId $applicationId -tenantName $tenantName
|
||||
|
||||
$stInfo = Get-ScenarioTesterInfo `
|
||||
-subscriptionId $subscriptionId `
|
||||
-resourceGroupName $resourceGroupName `
|
||||
-flowName $flowName `
|
||||
-scenarioTestKVBaseName $scenarioTestKVBaseName
|
||||
|
||||
$kvInfo = Get-KVKeyInfo -baseNamePrefix $kvBaseNamePrefix
|
||||
|
||||
$params = @{
|
||||
tenantId = $tenantId
|
||||
productName = $stInfo.productName
|
||||
applicationObjectId = $appAccInfo.objectId
|
||||
applicationId = $appAccInfo.applicationId
|
||||
applicationIdentifierUri = $appAccInfo.identifierUri
|
||||
authorityUri = $appAccInfo.authorityUri
|
||||
serviceUrl = $stInfo.serviceUrl
|
||||
secretKey = $appSecretKey
|
||||
kvServicesName = $stInfo.kvServicesName
|
||||
kvSparkName = $stInfo.kvSparkName
|
||||
blobUri = $stInfo.blobUri
|
||||
sparkStorageAccountName = $stInfo.sparkStorageAccountName
|
||||
eventHubName = $stInfo.eventHubName
|
||||
eventHubConnectionString = $stInfo.eventHubConnectionString
|
||||
isIotHub = $stInfo.isIotHub
|
||||
jobRunnerName = $stInfo.jobRunnerName
|
||||
databricksToken = $stInfo.databricksToken
|
||||
referenceDataUri = $stInfo.referenceDataUri
|
||||
udfSampleUri = $stInfo.udfSampleUri
|
||||
skipServerCertificateValidation = $skipServerCertificateValidation
|
||||
eventHubConnectionStringKVName = $stInfo.eventHubConnectionStringKVName
|
||||
referenceDataKVName = $stInfo.referenceDataKVName
|
||||
udfSampleKVName = $stInfo.udfSampleKVName
|
||||
databricksTokenKVName = $stInfo.databricksTokenKVName
|
||||
secretKeyKVName = $kvInfo.secretKeyKVName
|
||||
clientIdKVName = $kvInfo.clientIdKVName
|
||||
testClientId = $kvInfo.testClientId
|
||||
}
|
||||
|
||||
$templateName = $flowName
|
||||
|
||||
New-AzureRmResourceGroupDeployment `
|
||||
-Name "deployment-$templateName-$(get-date -f MM-dd-yyyy_HH_mm_ss)" `
|
||||
-ResourceGroupName $resourceGroupName `
|
||||
-TemplateFile $jobRunnerTemplateFile `
|
||||
-TemplateParameterObject $params `
|
||||
-Verbose `
|
||||
-ErrorAction stop
|
||||
|
|
@ -0,0 +1,271 @@
|
|||
# Stop execution if any exception happens at some step of the script
|
||||
$ErrorActionPreference = "stop"
|
||||
|
||||
#
|
||||
# Login
|
||||
#
|
||||
# Prompts the caller authentication in order to be granted access to the target subscription id and tenant id
|
||||
#
|
||||
function Login([string]$subscriptionId, [string]$tenantId)
|
||||
{
|
||||
# sign in
|
||||
Write-Host "Logging in..."
|
||||
|
||||
# select subscription
|
||||
Write-Host "Selecting subscription '$subscriptionId'"
|
||||
Write-Host "Checking credential..."
|
||||
Write-Host "Logging in for AzureRM"
|
||||
$azurermsub = Select-AzureRmSubscription -SubscriptionId $subscriptionId -ErrorAction SilentlyContinue
|
||||
if (!($azurermsub)) {
|
||||
Connect-AzureRmAccount -TenantId $tenantId
|
||||
$azurermsub = Select-AzureRmSubscription -SubscriptionId $subscriptionId
|
||||
}
|
||||
try {
|
||||
az account set --subscription $subscriptionId > $null 2>&1
|
||||
}
|
||||
catch {}
|
||||
|
||||
if (!$?) {
|
||||
Write-Host "Logging in for AzureCLI"
|
||||
az login --tenant $tenantId
|
||||
az account set --subscription $subscriptionId
|
||||
if (!$?)
|
||||
{
|
||||
Write-Host "Can't access to the subscription. Please run 'az login' to sign in and try again"
|
||||
Exit 40
|
||||
}
|
||||
}
|
||||
|
||||
Write-Host "Logging in for AzureAD"
|
||||
$acc = Connect-AzureAD -TenantId $tenantId
|
||||
$tenantName = $acc.Tenant.Domain
|
||||
$tenantId = $acc.Tenant.Id.Guid
|
||||
$userId = (az ad signed-in-user show --query 'objectId').Replace("""", "")
|
||||
|
||||
# Check login info
|
||||
if (!($tenantName) -or !($tenantId) -or !($userId)) {
|
||||
Write-Host "Error on getting tenantName, tenantId or userId"
|
||||
Exit 100
|
||||
}
|
||||
|
||||
# Output login info
|
||||
Write-Host "tenantId: " $tenantId
|
||||
Write-Host "tenantName: " $tenantName
|
||||
Write-Host "userId: " $userId
|
||||
|
||||
@{
|
||||
subscriptionId = $subscriptionId
|
||||
tenantId = $tenantId
|
||||
tenantName = $tenantName
|
||||
userId = $userId
|
||||
}
|
||||
}
|
||||
|
||||
#
|
||||
# Get-AppInfo
|
||||
#
|
||||
# Retrieves additional identifiers from a given application id and tenant such object, authorityuri and app name
|
||||
#
|
||||
function Get-AppInfo([string]$applicationId, [string]$tenantName)
|
||||
{
|
||||
$app = Get-AzureRmADApplication -ApplicationId $applicationId
|
||||
|
||||
if (!$app)
|
||||
{
|
||||
Write-Host "Error on getting the service application"
|
||||
Exit 30
|
||||
}
|
||||
|
||||
$applicationId = $app.ApplicationId
|
||||
|
||||
$servicePrincipal = Get-AzureRmADServicePrincipal -ApplicationId $applicationId
|
||||
if (!$servicePrincipal)
|
||||
{
|
||||
Write-Host "Error on getting the service application principal"
|
||||
Exit 30
|
||||
}
|
||||
|
||||
|
||||
$applicationId = $applicationId
|
||||
$objectId = $servicePrincipal.Id
|
||||
$identifierUri = $app.IdentifierUris[0]
|
||||
$authorityUri = "https://login.microsoftonline.com/$tenantName"
|
||||
$appName = $app.DisplayName
|
||||
|
||||
Write-Host "Application Id: $applicationId"
|
||||
Write-Host "Application name: $appName"
|
||||
Write-Host "Application Object Id: $objectId"
|
||||
Write-Host "Application identifier uri: $identifierUri"
|
||||
Write-Host "Authority Uri: $authorityUri"
|
||||
|
||||
@{
|
||||
applicationId = $applicationId
|
||||
appName = $appName
|
||||
objectId = $objectId
|
||||
identifierUri = $identifierUri
|
||||
authorityUri = $authorityUri
|
||||
}
|
||||
}
|
||||
|
||||
#
|
||||
# Get-ScenarioTesterInfo
|
||||
#
|
||||
# Gathers information of existing deployed resources from a previous data-accelerator deployment. Information from this step will be used
|
||||
# to generate the resources used by Scenario Tester Job Runner and are the initial parameters used to deploy data-accelerator.
|
||||
#
|
||||
function Get-ScenarioTesterInfo
|
||||
{
|
||||
param(
|
||||
[string]$subscriptionId,
|
||||
[string]$resourceGroupName,
|
||||
[string]$flowName,
|
||||
[string]$scenarioTestKVBaseName
|
||||
)
|
||||
$blobContainerName="samples"
|
||||
$blobPath="iotdevice/ScenarioTest.json"
|
||||
$eventHubConnectionStringSource="iotsample-input-eventhubconnectionstring"
|
||||
$databricksTokenSource="iotsample-info-databricksToken"
|
||||
$referenceDataSource="iotsample-referencedata-devicesdata"
|
||||
$udfSampleSource="iotsample-jarpath-udfsample"
|
||||
$isIotHub = "true"
|
||||
$eventHubType = "iothub"
|
||||
|
||||
$deployment = (Get-AzureRmResourceGroupDeployment -ResourceGroupName $resourceGroupName | where DeploymentName -Like 'deployment-Resource-Template.json*')
|
||||
|
||||
$productName = $deployment.Parameters.sites_web_name.Value
|
||||
|
||||
$kvSparkName = $deployment.Parameters.vaults_sparkKV_name.Value
|
||||
|
||||
$kvServicesName = $deployment.Parameters.vaults_servicesKV_name.Value
|
||||
|
||||
$sparkStorageAccountName = $deployment.Parameters.storageAccounts_spark_name.Value
|
||||
|
||||
$sparkType = $deployment.Parameters.sparkType.Value
|
||||
|
||||
$sfClusterName = $deployment.Parameters.clusterName.Value
|
||||
|
||||
$sfCluster = Get-AzureRmResource -ResourceGroupName $resourceGroupName -ResourceType Microsoft.ServiceFabric/clusters -Name $sfClusterName
|
||||
$sfClusterManageEndpoint = [uri]$sfCluster.Properties.managementEndpoint
|
||||
$sfClusterServiceUrl = $sfClusterManageEndpoint.scheme + "://" +$sfClusterManageEndpoint.Host
|
||||
Write-Host "SF Cluster management endpoint: $sfClusterServiceUrl"
|
||||
|
||||
$blobStorageAccount = Get-AzureRmResource -ResourceGroupName $resourceGroupName -ResourceType Microsoft.Storage/storageAccounts -Name $sparkStorageAccountName
|
||||
$blobPrimaryEndpoint = $blobStorageAccount.Properties.primaryEndpoints.blob
|
||||
$blobUri = $blobPrimaryEndpoint + $blobUriPath
|
||||
Write-Host "Blob uri path: $blobUri"
|
||||
|
||||
$eventHubConnectionString = (Get-AzureKeyVaultSecret -VaultName $kvSparkName -Name $eventHubConnectionStringSource).SecretValueText
|
||||
Write-Host "Event hub connection string: $eventHubConnectionString"
|
||||
|
||||
$csAsObject = $eventHubConnectionString -replace ";", "`n" | ConvertFrom-StringData
|
||||
$eventHubName = $csAsObject.EntityPath
|
||||
Write-Host "Event Hub name: $eventHubName"
|
||||
|
||||
$referenceDataUri = (Get-AzureKeyVaultSecret -VaultName $kvSparkName -Name $referenceDataSource).SecretValueText
|
||||
Write-Host "Reference data uri: $referenceDataUri"
|
||||
|
||||
$udfSampleUri = (Get-AzureKeyVaultSecret -VaultName $kvSparkName -Name $udfSampleSource).SecretValueText
|
||||
Write-Host "Udf sample uri: $udfSampleUri"
|
||||
|
||||
Write-Host "Spark type: $sparkType"
|
||||
$databricksToken = ""
|
||||
|
||||
if($sparkType.Equals("databricks"))
|
||||
{
|
||||
Write-Host "Fetching databricks token from secret"
|
||||
$databricksToken = (Get-AzureKeyVaultSecret -VaultName $kvSparkName -Name $databricksTokenSource).SecretValueText
|
||||
}
|
||||
else {
|
||||
Write-Host "Not fetching token, is not databricks"
|
||||
}
|
||||
|
||||
Write-Host "Databricks token: $databricksToken"
|
||||
|
||||
if($sparkType.Equals("databricks"))
|
||||
{
|
||||
$jsonFileName = "ScenarioTestDatabricks.json"
|
||||
}
|
||||
else
|
||||
{
|
||||
$jsonFileName = "ScenarioTestHDInsights.json"
|
||||
}
|
||||
|
||||
Write-Host "Json file name $jsonFileName"
|
||||
|
||||
if($sparkType.Equals("databricks"))
|
||||
{
|
||||
$kvSparkBase = "secretscope://$kvSparkName"
|
||||
}
|
||||
else
|
||||
{
|
||||
$kvSparkBase = "keyvault://$kvSparkName"
|
||||
}
|
||||
|
||||
$eventHubConnectionStringKVName = "${scenarioTestKVBaseName}-input-eventhubconnectionstring"
|
||||
$referenceDataKVName = "${scenarioTestKVBaseName}-referencedata-devicesdata"
|
||||
$udfSampleKVName = "${scenarioTestKVBaseName}-jarpath-udfsample"
|
||||
|
||||
$jsonInputFilePath = $PSScriptRoot + "\" + $jsonFileName
|
||||
$scenarioTestJson = Get-Content -Raw -Path $jsonInputFilePath
|
||||
$scenarioTestJson = $scenarioTestJson.Replace('$flowName', $flowName)
|
||||
$scenarioTestJson = $scenarioTestJson.Replace('$eventHubName', $eventHubName)
|
||||
$scenarioTestJson = $scenarioTestJson.Replace('$eventHubType', $eventHubType)
|
||||
$scenarioTestJson = $scenarioTestJson.Replace('$referenceDataUri', "$kvSparkBase/$referenceDataKVName")
|
||||
$scenarioTestJson = $scenarioTestJson.Replace('$udfSampleUri', "$kvSparkBase/$udfSampleKVName")
|
||||
$scenarioTestJson = $scenarioTestJson.Replace('$subscriptionId', $subscriptionId)
|
||||
$scenarioTestJson = $scenarioTestJson.Replace('$eventHubConnectionString', "$kvSparkBase/$eventHubConnectionStringKVName")
|
||||
if($sparkType.Equals("databricks"))
|
||||
{
|
||||
$scenarioTestJson = $scenarioTestJson.Replace('$databricksToken', $databricksToken)
|
||||
}
|
||||
$jsonOutputFilePath = $PSScriptRoot + "\ScenarioTest.json"
|
||||
Set-Content -Path $jsonOutputFilePath -Value $scenarioTestJson
|
||||
$storageAccountKey = (Get-AzureRmStorageAccountKey -ResourceGroupName $resourceGroupName -AccountName $sparkStorageAccountName).Value[0]
|
||||
Write-Host "Storage account Key: $storageAccountKey"
|
||||
$ctx = New-AzureStorageContext -StorageAccountName $sparkStorageAccountName -StorageAccountKey $storageAccountKey
|
||||
$blobUri = (Set-AzureStorageBlobContent -File "ScenarioTest.json" -Force `
|
||||
-Container $blobContainerName `
|
||||
-Blob $blobPath `
|
||||
-Context $ctx).ICloudBlob.uri.AbsoluteUri
|
||||
Write-Host "Blob uri: $blobUri"
|
||||
|
||||
@{
|
||||
productName = $productName
|
||||
blobUri = $blobUri
|
||||
eventHubName = $eventHubName
|
||||
eventHubConnectionString = $eventHubConnectionString
|
||||
isIotHub = $isIotHub
|
||||
serviceUrl = $sfClusterServiceUrl
|
||||
jobRunnerName = $productName + $flowName + $sparkType
|
||||
databricksToken = $databricksToken
|
||||
referenceDataUri = $referenceDataUri
|
||||
udfSampleUri = $udfSampleUri
|
||||
kvSparkName = $kvSparkName
|
||||
kvServicesName = $kvServicesName
|
||||
sparkStorageAccountName = $sparkStorageAccountName
|
||||
eventHubConnectionStringKVName = $eventHubConnectionStringKVName
|
||||
referenceDataKVName = $referenceDataKVName
|
||||
udfSampleKVName = $udfSampleKVName
|
||||
databricksTokenKVName = "${scenarioTestKVBaseName}-info-databricksToken"
|
||||
}
|
||||
}
|
||||
|
||||
#
|
||||
# Get-KVKeyInfo
|
||||
#
|
||||
# Utility function to easily retrieve secrets from a data-accelerator deployment given a base name prefix. This prefix is part of the initial parameters
|
||||
# of the previous data-accelerator deployment.
|
||||
#
|
||||
function Get-KVKeyInfo {
|
||||
param(
|
||||
[string]$baseNamePrefix
|
||||
)
|
||||
|
||||
return @{
|
||||
testClientId = "$baseNamePrefix-testClientId"
|
||||
secretKeyKVName = "$baseNamePrefix-scenarioTester-secretKey"
|
||||
clientIdKVName = "$baseNamePrefix-scenarioTester-clientId"
|
||||
}
|
||||
}
|
||||
|
||||
Export-ModuleMember -Function *
|
||||
Загрузка…
Ссылка в новой задаче