docker registry template embedded (#451)

This commit is contained in:
Javier Darsie 2019-09-30 18:08:11 -07:00 коммит произвёл vikasnav
Родитель 65a7a15145
Коммит ae9b7e764d
5 изменённых файлов: 303 добавлений и 263 удалений

Просмотреть файл

@ -11,7 +11,6 @@ Make sure you take care of the following pre-requisites before you start the set
- `Ubuntu Server 16.04-LTS` was syndicated from Azure Stack's Marketplace by the operator
- `Custom Script Extensions for Linux 2.0` was syndicated from Azure Stack's Marketplace by the operator
- You have access to a X.509 certificate in PFX format
- You can execute [htpasswd](https://httpd.apache.org/docs/2.4/programs/htpasswd.html) to generate the authorized users credentials
- You can [connect](https://docs.microsoft.com/en-us/azure-stack/user/azure-stack-powershell-configure-user) to the target Azure Stack instance using PowerShell
## Setup
@ -20,112 +19,17 @@ The following section details the required steps to perform before you deploy th
Once you went through the details, you should be able to tweak the [setup script](setup.ps1) and adjust it to your needs.
### Basic Authentication using .htpasswd files
`htpasswd` is a small command-line utility that creates and updates text files (usually named `.htpasswd`) used to store user credentials for basic HTTP authentication.
An usage example is shown below (add flag `-c` to create file `.htpasswd`):
```bash
htpasswd -Bb .htpasswd my-user my-password
```
#### Anonymous access
To allow anonymous access to the registry, update the `docker run` command executed by the [CSE script](script.sh) **before** you start the [storage configuration](#storage-configuration) step.
Deleting the lines that set variables REGISTRY_AUTH, REGISTRY_AUTH_HTPASSWD_PATH AND REGISTRY_AUTH_HTPASSWD_REALM will disable basic authentication.
### Storage configuration
The template instructs the container registry to use the [Azure storage driver](https://docs.docker.com/registry/storage-drivers/azure/) to persist the container images in a local storage account blob container.
We will also store the `.htpasswd` file in the same storage account to keep it secure and readily available when you need to upgrade your registry or guest OS to a new version.
### Key Vault configuration
You can use the PowerShell snipped below to automate the storage account setup process:
The deployment template will instruct Azure Resource Manager to drop your certificate in the virtual machine's file system. User credentials should be stored as secrets in the same local Key Vault instance where the PFX certificate is stored.
```powershell
# Set variables to match your environment
$location = "your-location"
$resourceGroup = "registry-rg"
$saName = "registry"
$saContainer = "images"
$tokenIni = Get-Date
$tokenEnd = $tokenIni.AddYears(1.0)
### Basic Authorization
# Create resource group
Write-Host "Creating resource group:" $resourceGroup
New-AzureRmResourceGroup -Name $resourceGroup -Location $location | out-null
# Create storage account
Write-Host "Creating storage account:" $saName
$sa = New-AzureRmStorageAccount -ResourceGroupName $resourceGroup -AccountName $saName -Location $location -SkuName Premium_LRS -EnableHttpsTrafficOnly 1
$saKey = (Get-AzureRmStorageAccountKey -ResourceGroupName $resourceGroup -AccountName $saName)[0].Value
# Create blob container
Write-Host "Creating blob container:" $saContainer
Set-AzureRmCurrentStorageAccount -ResourceGroupName $resourceGroup -AccountName $saName | out-null
$container = New-AzureStorageContainer -Name $saContainer
# Upload the CSE script so the template can later fetch it during deployment
Write-Host "Uploading configuration script"
Set-AzureStorageBlobContent -Container $saContainer -File script.sh | out-null
$cseToken = New-AzureStorageBlobSASToken -Container $saContainer -Blob "script.sh" -Permission r -StartTime $tokenIni -ExpiryTime $tokenEnd
$cseUrl = $container.CloudBlobContainer.Uri.AbsoluteUri + "/script.sh" + $cseToken
# The CSE script needs the .htpasswd file to configure the container registry
Write-Host "Uploading .htpasswd file"
Set-AzureStorageBlobContent -Container $saContainer -File .htpasswd | out-null
# Get htpasswd download URL
$htpasswdToken = New-AzureStorageBlobSASToken -Container $saContainer -Blob .htpasswd -Permission r -StartTime $tokenIni -ExpiryTime $tokenEnd
$htpasswdUrl = $container.CloudBlobContainer.Uri.AbsoluteUri + "/.htpasswd" + $htpasswdToken
```
## Key Vault configuration
The deployment template will instruct Azure Resource Manager to drop your certificate in the virtual machine's file system.
The snippet below creates the Key Vault resource and uploads the .pfx certificate.
```powershell
$kvName = "certs"
$secretName = "registry"
$pfxPath = "cert.pfx"
$pfxPass = ""
# Create key vault enabled for deployment
Write-Host "Creating key vault:" $kvName
$kv = New-AzureRmKeyVault -ResourceGroupName $resourceGroup -VaultName $kvName -Location $location -Sku standard -EnabledForDeployment
# Serialize certificate
$fileContentBytes = get-content $pfxPath -Encoding Byte
$fileContentEncoded = [System.Convert]::ToBase64String($fileContentBytes)
$jsonObject = @"
{
"data": "$filecontentencoded",
"dataType" :"pfx",
"password": "$pfxPass"
}
"@
$jsonObjectBytes = [System.Text.Encoding]::UTF8.GetBytes($jsonObject)
$jsonEncoded = [System.Convert]::ToBase64String($jsonObjectBytes)
$secret = ConvertTo-SecureString -String $jsonEncoded -AsPlainText -Force
# Upload certificate as secret
Write-Host "Storing certificate in key vault:" $pfxPath
$kvSecret = Set-AzureKeyVaultSecret -VaultName $kvName -Name $secretName -SecretValue $secret
```
### Certificate thumbprint
Your certificate thumbprint is a required parameter of the ARM template. The CSE script uses that information to find the certificate in the virtual machine' file system.
Run the following snipped to generate the certificate thumbprint:
```powershell
Write-Host "Computing certificate thumbprint"
$tp = Get-PfxCertificate -FilePath $pfxPath
```
User credentials should be stored as secrets in the same local Key Vault instance where the PFX certificate is stored. This can be achieved using the web UI or the SDK.
## Template deployment
@ -145,7 +49,7 @@ New-AzureRmResourceGroupDeployment `
### Upgrade
In order to upgrade the guest OS or the container registry itself, update [azuredeploy.json](azuredeploy.json) and/or [script.sh](script.sh) as needed and run once again `New-AzureRmResourceGroupDeployment` as previously indicated.
In order to upgrade the guest OS or the container registry itself, update [azuredeploy.json](azuredeploy.json) as needed and run once again `New-AzureRmResourceGroupDeployment` as previously indicated.
## Usage

Просмотреть файл

@ -50,10 +50,29 @@
"description": "The guest OS image version."
}
},
"storageAccountName": {
"pipName": {
"type": "string",
"metadata": {
"description": "An already existing storage account name."
"description": "The public IP resource name."
}
},
"pipDomainNameLabel": {
"type": "string",
"metadata": {
"description": "The public IP DNS label."
}
},
"pipAllocationMethod": {
"type": "string",
"defaultValue": "dynamic",
"metadata": {
"description": "The public IP allocation method."
}
},
"storageAccountResourceId": {
"type": "string",
"metadata": {
"description": "An already existing storage account resource identifier."
}
},
"storageAccountContainer": {
@ -62,31 +81,19 @@
"description": "An already existing storage account container name."
}
},
"storageAccountKey": {
"type": "string",
"metadata": {
"description": "The storage account access key."
}
},
"domainNameLabel": {
"type": "string",
"metadata": {
"description": "The public IP DNS label."
}
},
"keyVaultResourceId": {
"pfxKeyVaultResourceId": {
"type": "string",
"metadata": {
"description": "The Key Vault resource identifier."
}
},
"keyVaultSecretUrl": {
"pfxKeyVaultSecretUrl": {
"type": "string",
"metadata": {
"description": "Absolute URL to the Key Vault secret that stores the pfx certificate."
}
},
"certificateThumbprint": {
"pfxThumbprint": {
"type": "string",
"metadata": {
"description": "The certificate thumbprint."
@ -100,41 +107,47 @@
}
},
"registryReplicas": {
"type": "int",
"defaultValue": 20,
"type": "string",
"defaultValue": "20",
"metadata": {
"description": "Docker registry replicas."
}
},
"cseLocation": {
"type": "string",
"servicePrincipalClientId": {
"type": "securestring",
"metadata": {
"description": "Download URL of the script to be executed by the template custom script extension."
"description": "Client ID with access to list and get secrets from the credentials Key Vault instance"
}
},
"htpasswdLocation": {
"type": "string",
"servicePrincipalClientSecret": {
"type": "securestring",
"metadata": {
"description": "Download URL of the .htpasswd file."
"description": "Secret of the client with access to list and get secrets from the credentials Key Vault instance"
}
}
},
"variables": {
"vnetId": "[resourceId('Microsoft.Network/virtualNetworks', 'registry-vnet')]",
"rgname": "[resourceGroup().name]",
"nsgName": "[concat(variables('rgname'), '-nsg')]",
"nicName": "[concat(variables('rgname'), '-nic')]",
"vnetName": "[concat(variables('rgname'), '-vnet')]",
"vnetId": "[resourceId('Microsoft.Network/virtualNetworks',variables('vnetName'))]",
"subnetRef": "[concat(variables('vnetId'), '/subnets/default')]",
"tenantId": "[subscription().tenantId]",
"location": "[resourceGroup().location]",
"provisionScriptParameters": "[concat('ADMIN_USER_NAME=', parameters('adminUsername'),' REGISTRY_STORAGE_AZURE_ACCOUNTNAME=', parameters('storageAccountName'),' REGISTRY_STORAGE_AZURE_ACCOUNTKEY=', parameters('storageAccountKey'),' REGISTRY_STORAGE_AZURE_CONTAINER=', parameters('storageAccountContainer'),' CERT_THUMBPRINT=', parameters('certificateThumbprint'),' PIP_LABEL=', parameters('domainNameLabel'),' REGISTRY_TAG=', parameters('registryTag'),' REGISTRY_REPLICAS=', parameters('registryReplicas'))]"
"provisionScriptParameters": "[concat('ADMIN_USER_NAME=', parameters('adminUsername'),' SA_RESOURCE_ID=', parameters('storageAccountResourceId'),' SA_CONTAINER=', parameters('storageAccountContainer'),' KV_RESOURCE_ID=', parameters('pfxKeyVaultResourceId'),' CERT_THUMBPRINT=', parameters('pfxThumbprint'),' PIP_LABEL=', parameters('pipDomainNameLabel'),' REGISTRY_TAG=', parameters('registryTag'),' SPN_CLIENT_ID=',parameters('servicePrincipalClientId'),' SPN_CLIENT_SECRET=',parameters('servicePrincipalClientSecret'),' REGISTRY_REPLICAS=', parameters('registryReplicas'))]"
},
"resources": [
{
"type": "Microsoft.Compute/virtualMachines",
"name": "registry-vm",
"name": "[concat(variables('rgname'),'-vm')]",
"apiVersion": "2017-03-30",
"location": "[resourceGroup().location]",
"properties": {
"osProfile": {
"computerName": "registry",
"computerName": "[concat(variables('rgname'),'-vm')]",
"adminUsername": "[parameters('adminUsername')]",
"customData": "[base64(concat('#cloud-config\n\nwrite_files:\n- path: \"/opt/azure/containers/script.sh\"\n permissions: \"0744\"\n encoding: gzip\n owner: \"root\"\n content: !!binary |\n H4sIAAAAAAAA/+w6/XPbtpK/86/Y8HFqOzMkJTvOm1Me06MlxtFElnwinb6em+PAJCSxJgkaAO2osv73G4CfouSmXzO9H66dcejF7mK/dwH4H6/Muyg17xBbgf5VUZz53LevPX88dT17MvG98ZUzu/Gs/4Dmv3+AFyWY5ByilHEUx1G6BIof8ojiEFDGIUPBPVpiJtldjV13PL30h3PP/zCeOFa/B7DD7gKFEGDKga/y5C6jUcphNods8RVSInaBe7yGR5THEs5xksWIY0giFpB0ES1zsfHnK2A4oJgz8S+PSLqz/Sfnx3L7/t+x/Y3rzP3h3Bk5U29sT1yrf1puPyWQM0whoDjEKY9QzEpOsCB5GgJJYRk94pYYkvPcuRy73vxHfzrz/PnNdDqeXlr9s7abVhgCknIUpZgCxcuIcbqGBYpiHAInwDiiHFgeBJixRR7H60Lm2cWPMgomY9erQ+D0fD8AnlDEhfcXhEJC7tbS+4zkNKic7/qX14XxR7MfppOZPWoYvn2J1ZULl9eXUt+QPKUxQWEjWDc0T/95IDL3BCtDtWDjePbI9mzrbDcSGzYOpYTCQ47pWnBJMEch4qhS6Xo+G/kj52JfpzenLwix4jxjA9OsUsNIooASRhbcCEhiFoFk5nd5ynOz/9bovalR9RpVzygJjRDf7Qly/enSt0cj/4M9nlhvzko1PtSeRmEIFGcEsvslLKIY15l+cz2yPadJ9MOZ3lYFZVxfYg55Foo04AQCkmQx5lhRKOZ0HSShHy18EWc5xccnsFEEN7EWYWZp/XeSn89ijDNLO30HvNjG0s7egcRlq2jB4bvvOh9yTcgQibTUjhl+gD5oJeeTdxASpRK+5Ala/bH5z63gdEcxuofnZ/ipxo0WcAtaBDp+qLnBl3fAVzhV2qFBMc9pCv0aiGOGdzCkUqA1Ctari0h+hiTFylaRCAtCfZRxPybBPasN9bSKYgwLWRXMR0TNOLozw+x+aQq8BoQybsYR46wFD1CwwnIF0WAVPeJy8b0Z4kczzeMYTt9/19+xFA5WBI5+aLmY4hgjhoEsZEZL8Y6UXRXP2toIJZaY+0VM7Lm835O/CqwCwyc5z3JumTzJzDKg9GLJIPnv8vO+IeslYTPQ9bpKg47qtSqK9QXoTXmoVl/BcSfMhdXgGTjGoO3pAc+wpDgD3QH1f45vf3C+DIzXJ8/Ht9j5Qqnx+kRTT0ToNREXIH6Iz+HwPIz8F0Vv6c7zg3H6goEbh5eG+7uS/E86v5QedAKj7H45GMwy0bnZYGCpur4gNMCSAYlDFXQ9JXpJoVMckCTBachE+PzdpaVKriYFf583S61GOGuq0M3FzdS78efOxLFdx9KOY3bnV4VBp6CzE4knm/JnZ+6OZ1NLPTN6xltVqWOh0wygf9qDczg9hyCn8e/ritpmV6StKduhqIDwHmQl6XRKufT8DPhrxEF7cbR5WdgewLn4GWQv8zcxD2TBLeceCTVC809t/HusdI/XrJHMQCwQ5SgTwR9iRBNC98xjiOVd+V4e1P6AeST/2jCc5ozjUAD37fKNfXejepd2f3yRJDt0VYaf9uCsJ+2KMtEiT/WcRzGTw6GO02WUYkvbtGN5W6wFcbS3oOsojsmTLobTJUUhZgec3ZlTpVSipSckBB1dQkiCe0xB29ijq/G0OCBM7Stn2yhRTlt+6XhRDmVA/PzQKoLRAl7tqSt0hbe9XtE2SvpO0fmZ5DRFccDjorRlaIkp6PkOTVONdsx+SLnuhLPAPFjN7Jyv6priTEfXs/HUcy3tOLkXJ6mihkitdAa6LuMMzqsvPcQxWosY0/UEfdVFD4G3PdG1mxJb5UeCUrTECU65oW0+/NdoujWryd3EaZiRKOXse5RF+iOmLCKpddrrn+u9vt7rw3vQNrV426KAFbX7e9BTDL29ot3yd3mgUFpGkD9m9o330dKOf34QFdNAuWDAowCJLmPEZBmlTinYzvaFVeRcpm0kk209YmBQTRQumKZ2hMQPB4T0Zp+cqX8zn1hqxckkQo5Tk5N7nKrKXqs5QKJtPGdqTz1/PDpEvogqdw+bQ2zt9Lnjzm7mQ+dFO6A8jHAaYHbb+7JjBngGhkNQ2fMBF9cweSQ2ntWTwh5SfEs7/iMxpf8brmeu14ot/SOoQ5JynHLdW2d4ACjL4lJy86v+9PQk5oREz2mM04CEOFTb5CGoS4pS7vN1hq0gjnDK/dZRv4tcYkShpW3c66k/nIydwu47mLoI6mbPmq64ONildZ3h3PG+QU9x0b4sbVP5a4dC29RhIdxSOPLIQPLqwJexcFQ64NPnIngq/2ibT5+LymYUrqqSs7wvKWKokNL9g45rBFXldkLMbqa/1fs9vddXpU9FWSI0+kX6cQAXGFFZjKWWW7VW0XhEcY5vvxhRWKpHEzBWPEOMPYUSwEkerDowUbnLSxwxw25K7bZlfYRa3GJBmqcZorRNQWuaWqmLeojA0o7LEnGAi/BSkHMRVEfmkbDR6UmXyWd7cuP8aYuXVi9E/iutftKq8IVtQb+4ayy9q/i2+VXqtW16UVMnXwk1GwZ7JV2YU23WF+IgHjHAScbXRuOEclbXfu2Gr1MZXU4oWuJPeP1XVMb/r3X/N2qda+/WugMjiLZxbb/aRzZQcUAQcdBNlX+WqVJx/uT8+Jf5VpViyFLymzKxHQsTnC75agC9JkFVQ5w6bntfikxVT5StosjsmcyGtjeeTQdNcdA2FXBb4NTDxKCF00wYBVJnLB5IpL1ZWaLuGnhQ8OtYvcYczqaePZ4680G1cxtY4n36fIDjLrDEHDpzz/c+3lxdXM/HU68QswMsUa/H176IiR3TVMAWzsS+cCaDDo4Elkj11b9nX9ZatIFdvLlzPRkPbXfQxquAlWnaGTioTLOTlnuIRboN4PXr14ri2t2utOuBeoIlKzHFHt3eDlCc5sngy5fX2tGJUk4JDX3H3t+iV5x/e/5odmWPpwUjtbGuaWq1EQ2tCkcjiEkeoiwztqoi0KxWqBrapsNwqyj1I5a672QjoFxV6nemQxgZfVSVwmiWdkwynDIWA0VpCPodYvjtGzg7PVEU2av0BZgk4yb6JafYfFyF1eWTUd237/SvonehewaCE0SJPPGx+yjLonQJIc5wGuI0iDCr7r1k8qvVVKPTBZgrkmB50YBp0T9xLO+hW6AlJXkW4riC1YeH1i2SsojKevCI4ihEvCuCUrXjBaj1hfYTQkucclPbVHbeqrs6Nkeuzqui3PDXeVae+SbPCrFRAoWhUCDAlDPgRBBDgHTGCcVKkB3Ybogpjxaid2JmZDhRQTVzRs2YBCg22QpRbAZID1poJvqFtciKcCruPfQOailWQLJ1+Xgo3y8JJCRPOcgDpeJ6s7ljqfIahrHYlNKb1TOgqiT3YURBz0CTmIf12HGFaCICdbsDfomwZe82YQMutVhiLsOj6Z1Qn9ZblwgltgQIdFYMVHCP13sTVhe3+8CqfPSubdf9YeSPxnNLNSkh3BSzV9sqbZy9061SjIBDEmJL+15G3q283y2B8MoSp/H9QGtQZHgFWXuebW9pNieKQhv5Wlu8tZcvufXjrhIgDv/6lzP7AO/flzdLuqgShGFjncRKOWUM4OjsSGGYPkYBZgMFal6DIoNF0Rg0wG5XKSpCFpMSHyAhoSQo5kQcKtWALAHshYZTIUmV/IzEUVBzBKFWGBVzCUn18raxudcXU88Azoub/oxQzipSHdQ3b84G571er6hqjyTOE9xa302GQedXShrEOiQG8mfhwfQxoiQVIdqwrNWbzC59ezh0XFd8jcaufTFxRtYCNZcrLWyRD/alY8nq/uKyb//3zdwRbGc3U69oj5uy0W5/I5UcIzfFPPlNmnoQsrpz0TcI5449ubLKoXcfWV6E7RyR95b9Ovqvbe+jJc1uGr+VpthfnVe5MccoTtR9qo+ed+3bo9Hc6hnyfxkuL+BVrbo8WB7QS6J5E1cO6y+U2nbR+xUOYloYfxgPbc/5FU513VWc2QdFKW+R2ROiCURpxGsIR8F9maxNxdCDQ9VBKV90ewrLQ1LdTLM14ziBjOYpBh2Jtvpdp6kT0RTrvzBhHPGcKcPxyNKOSyYZq8Y2tRLCrz6M/k+GOFKsMAqFZP3TE8X1bO/GbeijlGU44GKmLkbAnx9ANcTBw+WiSbpyT8HlK6JLdlJW4leiqQlOYFmg0jxNo3Sp1vUYOp3/0N/SiPL8vwEAAP//y1cPrx8lAAA='))]",
"linuxConfiguration": {
"disablePasswordAuthentication": true,
"ssh": {
@ -149,11 +162,11 @@
"secrets": [
{
"sourceVault": {
"id": "[parameters('keyVaultResourceId')]"
"id": "[parameters('pfxKeyVaultResourceId')]"
},
"vaultCertificates": [
{
"certificateUrl": "[parameters('keyVaultSecretUrl')]"
"certificateUrl": "[parameters('pfxKeyVaultSecretUrl')]"
}
]
}
@ -181,29 +194,25 @@
"networkProfile": {
"networkInterfaces": [
{
"id": "[resourceId('Microsoft.Network/networkInterfaces', 'registry-nic')]"
"id": "[resourceId('Microsoft.Network/networkInterfaces',variables('nicName'))]"
}
]
}
},
"dependsOn": [
"[concat('Microsoft.Network/networkInterfaces/', 'registry-nic')]"
"[concat('Microsoft.Network/networkInterfaces/',variables('nicName'))]"
]
},
{
"apiVersion": "2017-03-30",
"dependsOn": [
"[concat('Microsoft.Compute/virtualMachines/', 'registry-vm')]"
"[concat('Microsoft.Compute/virtualMachines/',variables('rgname'),'-vm')]"
],
"location": "[resourceGroup().location]",
"name": "registry-vm/cse",
"name": "[concat(variables('rgname'),'-vm/cse')]",
"properties": {
"protectedSettings": {
"fileUris": [
"[parameters('cseLocation')]",
"[parameters('htpasswdLocation')]"
],
"commandToExecute": "[concat(variables('provisionScriptParameters'),' LOCATION=',variables('location'),' FQDN=', '\"', reference(resourceId('Microsoft.Network/publicIPAddresses', 'registry-pip'),'2017-10-01').dnsSettings.fqdn,'\"',' ./script.sh >> /var/log/azure/docker-registry.log 2>&1')]"
"commandToExecute": "[concat(variables('provisionScriptParameters'),' LOCATION=',variables('location'),' TENANT_ID=',variables('tenantId'),' PIP_FQDN=', '\"', reference(resourceId('Microsoft.Network/publicIPAddresses',parameters('pipName')),'2017-10-01').dnsSettings.fqdn,'\"',' /opt/azure/containers/script.sh >> /var/log/azure/docker-registry.log 2>&1')]"
},
"publisher": "Microsoft.Azure.Extensions",
"settings": {},
@ -215,7 +224,7 @@
},
{
"type": "Microsoft.Network/virtualNetworks",
"name": "registry-vnet",
"name": "[concat(variables('rgname'),'-vnet')]",
"apiVersion": "2017-10-01",
"location": "[resourceGroup().location]",
"properties": {
@ -236,7 +245,7 @@
},
{
"type": "Microsoft.Network/networkInterfaces",
"name": "registry-nic",
"name": "[variables('nicName')]",
"apiVersion": "2017-10-01",
"location": "[resourceGroup().location]",
"properties": {
@ -249,19 +258,19 @@
},
"privateIPAllocationMethod": "Dynamic",
"publicIpAddress": {
"id": "[resourceId('Microsoft.Network/publicIpAddresses', 'registry-pip')]"
"id": "[resourceId('Microsoft.Network/publicIpAddresses',parameters('pipName'))]"
}
}
}
],
"networkSecurityGroup": {
"id": "[resourceId('Microsoft.Network/networkSecurityGroups', 'registry-nsg')]"
"id": "[resourceId('Microsoft.Network/networkSecurityGroups',variables('nsgName'))]"
}
},
"dependsOn": [
"[concat('Microsoft.Network/virtualNetworks/', 'registry-vnet')]",
"[concat('Microsoft.Network/publicIpAddresses/', 'registry-pip')]",
"[concat('Microsoft.Network/networkSecurityGroups/', 'registry-nsg')]"
"[concat('Microsoft.Network/virtualNetworks/',variables('rgname'),'-vnet')]",
"[concat('Microsoft.Network/publicIpAddresses/',parameters('pipName'))]",
"[concat('Microsoft.Network/networkSecurityGroups/',variables('nsgName'))]"
]
},
{
@ -269,19 +278,19 @@
"sku": {
"name": "Basic"
},
"name": "registry-pip",
"name": "[parameters('pipName')]",
"apiVersion": "2017-10-01",
"location": "[resourceGroup().location]",
"properties": {
"publicIpAllocationMethod": "Dynamic",
"publicIpAllocationMethod": "[parameters('pipAllocationMethod')]",
"dnsSettings": {
"domainNameLabel": "[parameters('domainNameLabel')]"
"domainNameLabel": "[parameters('pipDomainNameLabel')]"
}
}
},
{
"type": "Microsoft.Network/networkSecurityGroups",
"name": "registry-nsg",
"name": "[variables('nsgName')]",
"apiVersion": "2017-10-01",
"location": "[resourceGroup().location]",
"properties": {

Просмотреть файл

@ -2,34 +2,61 @@
"schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"storageAccountName": {
"value": ""
},
"storageAccountContainer": {
"value": ""
},
"storageAccountKey": {
"value": ""
},
"keyVaultResourceId": {
"value": ""
},
"keyVaultSecretUrl": {
"value": ""
},
"certificateThumbprint": {
"adminUsername": {
"value": ""
},
"adminPublicKey": {
"value": ""
},
"domainNameLabel": {
"virtualMachineSize": {
"value": ""
},
"cseLocation": {
"virtualMachinePublisher": {
"value": ""
},
"htpasswdLocation": {
"virtualMachineOffer": {
"value": ""
},
"virtualMachineSku": {
"value": ""
},
"virtualMachineVersion": {
"value": ""
},
"pipName": {
"value": ""
},
"pipDomainNameLabel": {
"value": ""
},
"pipAllocationMethod": {
"value": ""
},
"storageAccountResourceId": {
"value": ""
},
"storageAccountContainer": {
"value": ""
},
"pfxKeyVaultResourceId": {
"value": ""
},
"pfxKeyVaultSecretUrl": {
"value": ""
},
"pfxThumbprint": {
"value": ""
},
"registryTag": {
"value": ""
},
"registryReplicas": {
"value": ""
},
"servicePrincipalClientId": {
"value": ""
},
"servicePrincipalClientSecret": {
"value": ""
}
}

Просмотреть файл

@ -1,18 +1,25 @@
#!/bin/bash -x
#!/bin/bash
#
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT license.
ERR_APT_INSTALL_TIMEOUT=9 # Timeout installing required apt packages
ERR_MISSING_CRT_FILE=10 # Bad cert thumbprint OR pfx not in key vault OR template misconfigured VM secrets section
ERR_MISSING_KEY_FILE=11 # Bad cert thumbprint OR pfx not in key vault OR template misconfigured VM secrets section
ERR_REGISTRY_NOT_RUNNING=13 # the container registry failed to start successfully
ERR_MOBY_APT_LIST_TIMEOUT=25 # Timeout waiting for moby apt sources
ERR_MS_GPG_KEY_DOWNLOAD_TIMEOUT=26 # Timeout waiting for MS GPG key download
ERR_MOBY_INSTALL_TIMEOUT=27 # Timeout waiting for moby install
# ===
# NOTE: the CSE script is embedded in azuredeploy.json (base64 encoding)
# Update the VM resource (.properties.osPrifile.customData) to alter the CSE script.
# ===
ERR_APT_INSTALL_TIMEOUT=9 # Timeout installing required apt packages
ERR_MISSING_CRT_FILE=10 # Bad cert thumbprint OR pfx not in key vault OR template misconfigured VM secrets section
ERR_MISSING_KEY_FILE=11 # Bad cert thumbprint OR pfx not in key vault OR template misconfigured VM secrets section
ERR_MISSING_USER_CREDENTIALS=12 # No user credentials secret found on given key vault
ERR_REGISTRY_NOT_RUNNING=13 # The container registry failed to start successfully
ERR_MOBY_APT_LIST_TIMEOUT=25 # Timeout waiting for moby apt sources
ERR_MS_GPG_KEY_DOWNLOAD_TIMEOUT=26 # Timeout waiting for MS GPG key download
ERR_MOBY_INSTALL_TIMEOUT=27 # Timeout waiting for moby install
ERR_METADATA=30 # Error querying metadata
ERR_MS_PROD_DEB_DOWNLOAD_TIMEOUT=42 # Timeout waiting for https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb
ERR_MS_PROD_DEB_PKG_ADD_FAIL=43 # Failed to add repo pkg file
ERR_APT_UPDATE_TIMEOUT=99 # Timeout waiting for apt-get update to complete
UBUNTU_RELEASE=$(lsb_release -r -s)
MOBY_VERSION="3.0.6"
ERR_MS_PROD_DEB_PKG_ADD_FAIL=43 # Failed to add repo pkg file
ERR_APT_UPDATE_TIMEOUT=99 # Timeout waiting for apt-get update to complete
retrycmd_if_failure() {
retries=$1; wait_sleep=$2; timeout=$3;
@ -66,13 +73,16 @@ apt_get_install() {
wait_for_apt_locks
}
installDeps() {
UBUNTU_RELEASE=$(lsb_release -r -s)
MOBY_VERSION="3.0.6"
retrycmd_if_failure 120 5 25 curl https://packages.microsoft.com/config/ubuntu/${UBUNTU_RELEASE}/prod.list > /tmp/microsoft-prod.list || exit $ERR_MOBY_APT_LIST_TIMEOUT
retrycmd_if_failure 10 5 10 cp /tmp/microsoft-prod.list /etc/apt/sources.list.d/ || exit $ERR_MOBY_APT_LIST_TIMEOUT
retrycmd_if_failure 120 5 25 curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > /tmp/microsoft.gpg || exit $ERR_MS_GPG_KEY_DOWNLOAD_TIMEOUT
retrycmd_if_failure 10 5 10 cp /tmp/microsoft.gpg /etc/apt/trusted.gpg.d/ || exit $ERR_MS_GPG_KEY_DOWNLOAD_TIMEOUT
apt_get_update || exit $ERR_APT_UPDATE_TIMEOUT
apt_get_install 20 30 120 moby-engine=${MOBY_VERSION} moby-cli=${MOBY_VERSION} --allow-downgrades || exit $ERR_MOBY_INSTALL_TIMEOUT
apt_get_install 20 30 120 apache2-utils moby-engine=${MOBY_VERSION} moby-cli=${MOBY_VERSION} --allow-downgrades || exit $ERR_MOBY_INSTALL_TIMEOUT
usermod -aG docker ${ADMIN_USER_NAME}
for apt_package in curl jq; do
@ -82,26 +92,97 @@ installDeps() {
fi
done
}
fetchOAuth() {
ENDPOINTS=$(mktemp)
curl -s --retry 5 --retry-delay 10 --max-time 60 -f \
https://management.${FQDN}/metadata/endpoints?api-version=2015-01-01 > ${ENDPOINTS}
echo ADMIN_USER_NAME: ${ADMIN_USER_NAME}
echo REGISTRY_STORAGE_AZURE_ACCOUNTNAME: ${REGISTRY_STORAGE_AZURE_ACCOUNTNAME}
echo REGISTRY_STORAGE_AZURE_ACCOUNTKEY: ${REGISTRY_STORAGE_AZURE_ACCOUNTKEY}
echo REGISTRY_STORAGE_AZURE_CONTAINER: ${REGISTRY_STORAGE_AZURE_CONTAINER}
echo CERT_THUMBPRINT: ${CERT_THUMBPRINT}
echo FQDN: ${FQDN}
echo LOCATION: ${LOCATION}
echo PIP_LABEL: ${PIP_LABEL}
echo REGISTRY_TAG: ${REGISTRY_TAG}
echo REGISTRY_REPLICAS: ${REGISTRY_REPLICAS}
if [ $? -ne 0 ]; then
exit $ERR_METADATA
fi
OAUTH=$(jq -r .authentication.loginEndpoint ${ENDPOINTS})
echo ${OAUTH} | grep -e "/adfs$"
if [ $? -eq 0 ]; then
TOKEN_URL="${OAUTH}/oauth2/token"
else
TOKEN_URL="${OAUTH}${TENANT_ID}/oauth2/token"
fi
}
fetchCredentials() {
RESOURCE=$(jq -r .authentication.audiences[0] ${ENDPOINTS} | sed "s|https://management.|https://vault.|")
TOKEN=$(curl -s --retry 5 --retry-delay 10 --max-time 60 -f -X POST \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials" \
-d "client_id=${SPN_CLIENT_ID}" \
--data-urlencode "client_secret=${SPN_CLIENT_SECRET}" \
--data-urlencode "resource=${RESOURCE}" \
${TOKEN_URL} | jq -r '.access_token')
KV_URL="https://${KV_NAME}.vault.${FQDN}/secrets"
SECRETS=$(curl -s --retry 5 --retry-delay 10 --max-time 60 -f \
"${KV_URL}?api-version=2016-10-01" -H "Authorization: Bearer ${TOKEN}" | jq -r .value[].id)
rm .htpasswd
touch .htpasswd
for secret in ${SECRETS}
do
SECRET_NAME_VERSION="${secret//$KV_URL}"
SECRET_NAME=$(echo ${SECRET_NAME_VERSION} | cut -d '/' -f 2)
SECRET_VALUE=$(curl -s --retry 5 --retry-delay 10 --max-time 60 -f \
"${secret}?api-version=2016-10-01" -H "Authorization: Bearer ${TOKEN}" | jq -r .value)
htpasswd -Bb .htpasswd ${SECRET_NAME} ${SECRET_VALUE}
done
if [ ! -s .htpasswd ]; then
echo "file .htpasswd is empty, credentials were not created or there was an error fetching credentials from keyvault"
exit $ERR_MISSING_USER_CREDENTIALS
fi
}
fetchStorageKeys() {
RESOURCE=$(jq -r .authentication.audiences[0] ${ENDPOINTS})
TOKEN=$(curl -s --retry 5 --retry-delay 10 --max-time 60 -f -X POST \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials" \
-d "client_id=${SPN_CLIENT_ID}" \
--data-urlencode "client_secret=${SPN_CLIENT_SECRET}" \
--data-urlencode "resource=${RESOURCE}" \
${TOKEN_URL} | jq -r '.access_token')
SA_URL="https://management.${FQDN}/${SA_RESOURCE_ID}/listKeys?api-version=2017-10-01"
SA_KEY=$(curl -s --retry 5 --retry-delay 10 --max-time 60 -f -X POST \
"${SA_URL}" -H "Authorization: Bearer ${TOKEN}" -H "Content-Length: 0" | jq -r ".keys[0].value")
}
echo LOCATION: ${LOCATION}
echo TENANT_ID: ${TENANT_ID}
echo ADMIN_USER_NAME: ${ADMIN_USER_NAME}
echo SA_RESOURCE_ID: ${SA_RESOURCE_ID}
echo SA_CONTAINER: ${SA_CONTAINER}
echo KV_RESOURCE_ID: ${KV_RESOURCE_ID}
echo CERT_THUMBPRINT: ${CERT_THUMBPRINT}
echo PIP_FQDN: ${PIP_FQDN}
echo PIP_LABEL: ${PIP_LABEL}
echo REGISTRY_TAG: ${REGISTRY_TAG}
echo REGISTRY_REPLICAS: ${REGISTRY_REPLICAS}
echo SPN_CLIENT_ID: ${SPN_CLIENT_ID}
echo SPN_CLIENT_SECRET: ***
SA_NAME=$(echo ${SA_RESOURCE_ID} | grep -oh -e '[[:alnum:]]*$')
KV_NAME=$(echo ${KV_RESOURCE_ID} | grep -oh -e '[[:alnum:]]*$')
EXT_DOMAIN_NAME="${PIP_FQDN//$PIP_LABEL.$LOCATION.cloudapp.}"
FQDN=${LOCATION}.${EXT_DOMAIN_NAME}
EXTERNAL_FQDN="${FQDN//$PIP_LABEL.$LOCATION.cloudapp.}"
REGISTRY_STORAGE_AZURE_REALM=${LOCATION}.${EXTERNAL_FQDN}
CRT_FILE="${CERT_THUMBPRINT}.crt"
KEY_FILE="${CERT_THUMBPRINT}.prv"
SECRET=$(openssl rand -base64 32)
if [ -f /var/log.vhd/azure/golden-image-install.complete ]; then
echo "golden image; skipping dependencies installation"
if [ -f /opt/azure/vhd-install.complete ]; then
echo "aks base image; skipping dependencies installation"
rm -rf /home/packer
deluser packer
groupdel packer
@ -119,8 +200,7 @@ if [ ! -f "/var/lib/waagent/${KEY_FILE}" ]; then
fi
echo adding certs to the ca-store
CRT_DST_PATH="/usr/local/share/ca-certificates"
cp "/var/lib/waagent/Certificates.pem" "${CRT_DST_PATH}/azsCertificate.crt"
cp "/var/lib/waagent/Certificates.pem" "/usr/local/share/ca-certificates/azsCertificate.crt"
update-ca-certificates
echo copy user cert to mount point
@ -129,11 +209,17 @@ mkdir -p $STORE
cp "/var/lib/waagent/${CRT_FILE}" "${STORE}/${CRT_FILE}"
cp "/var/lib/waagent/${KEY_FILE}" "${STORE}/${KEY_FILE}"
echo moving .htpasswd to mount point
echo getting management endpoints
fetchOAuth
echo fetching storage key
fetchStorageKeys
echo fetching user credentials
HTPASSWD_DIR="/root/auth"
mkdir -p $HTPASSWD_DIR
awk '{ sub("\r$", ""); print }' .htpasswd > .htpasswd.tmp
cp .htpasswd.tmp $HTPASSWD_DIR/.htpasswd
fetchCredentials
cp .htpasswd $HTPASSWD_DIR/.htpasswd
echo starting registry container
cat <<EOF >> docker-compose.yml
@ -146,18 +232,19 @@ services:
replicas: ${REGISTRY_REPLICAS}
restart_policy:
condition: on-failure
delay: 5s
delay: 5s
ports:
- "443:5000"
volumes:
- /etc/ssl/certs:/etc/ssl/certs:ro
- /root/auth:/auth
environment:
- REGISTRY_LOG_ACCESSLOG_DISABLED=false
- REGISTRY_STORAGE=azure
- REGISTRY_STORAGE_AZURE_ACCOUNTNAME=${REGISTRY_STORAGE_AZURE_ACCOUNTNAME}
- REGISTRY_STORAGE_AZURE_ACCOUNTKEY=${REGISTRY_STORAGE_AZURE_ACCOUNTKEY}
- REGISTRY_STORAGE_AZURE_CONTAINER=${REGISTRY_STORAGE_AZURE_CONTAINER}
- REGISTRY_STORAGE_AZURE_REALM=${REGISTRY_STORAGE_AZURE_REALM}
- REGISTRY_STORAGE_AZURE_ACCOUNTNAME=${SA_NAME}
- REGISTRY_STORAGE_AZURE_ACCOUNTKEY=${SA_KEY}
- REGISTRY_STORAGE_AZURE_CONTAINER=${SA_CONTAINER}
- REGISTRY_STORAGE_AZURE_REALM=${FQDN}
- REGISTRY_AUTH=htpasswd
- REGISTRY_AUTH_HTPASSWD_PATH=/auth/.htpasswd
- REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm"

Просмотреть файл

@ -1,24 +1,25 @@
function Random-Name {
Param ([int]$length)
-join ((97..122) | Get-Random -Count $length | % {[char]$_})
}
# Set variables to match your environment
#########################################
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT license.
$location = ""
$resourceGroup = ""
$saName = Random-Name 10
$saContainer = Random-Name 10
$tokenIni = Get-Date
$tokenEnd = $tokenIni.AddYears(1.0)
$saName = ""
$saContainer = ""
$kvName = Random-Name 10
$secretName = Random-Name 10
$kvName = ""
$pfxSecret = ""
$pfxPath = ""
$pfxPass = ""
$dnsSubDomain = ""
$spnName = ""
$spnSecret = ""
$userName = ""
$userPass = ""
$dnsLabelName = ""
$sshKey = ""
$vmSize = ""
$registryTag = "2.7.1"
$registryReplicas = "5"
# RESOURCE GROUP
# =============================================
@ -34,25 +35,16 @@ New-AzureRmResourceGroup -Name $resourceGroup -Location $location | out-null
# Create storage account
Write-Host "Creating storage account:" $saName
$sa = New-AzureRmStorageAccount -ResourceGroupName $resourceGroup -AccountName $saName -Location $location -SkuName Premium_LRS -EnableHttpsTrafficOnly 1
$saKey = (Get-AzureRmStorageAccountKey -ResourceGroupName $resourceGroup -AccountName $saName)[0].Value
# Create container
Write-Host "Creating blob container:" $saContainer
Set-AzureRmCurrentStorageAccount -ResourceGroupName $resourceGroup -AccountName $saName | out-null
$container = New-AzureStorageContainer -Name $saContainer
New-AzureStorageContainer -Name $saContainer | out-null
# Upload configuration script
Write-Host "Uploading configuration script"
Set-AzureStorageBlobContent -Container $saContainer -File script.sh | out-null
$cseToken = New-AzureStorageBlobSASToken -Container $saContainer -Blob "script.sh" -Permission r -StartTime $tokenIni -ExpiryTime $tokenEnd
$cseUrl = $container.CloudBlobContainer.Uri.AbsoluteUri + "/script.sh" + $cseToken
# Upload htpasswd
Write-Host "Uploading htpasswd file"
Set-AzureStorageBlobContent -Container $saContainer -File .htpasswd | out-null
$htpasswdToken = New-AzureStorageBlobSASToken -Container $saContainer -Blob .htpasswd -Permission r -StartTime $tokenIni -ExpiryTime $tokenEnd
$htpasswdUrl = $container.CloudBlobContainer.Uri.AbsoluteUri + "/.htpasswd" + $htpasswdToken
Write-Host "=> Storage Account Resource ID:" $sa.Id
Write-Host "Assigning contributor role to" $spnName
New-AzureRMRoleAssignment -ApplicationId $spnName -RoleDefinitionName "Contributor" -Scope $sa.Id
# KEY VAULT
# =============================================
@ -60,8 +52,13 @@ $htpasswdUrl = $container.CloudBlobContainer.Uri.AbsoluteUri + "/.htpasswd" + $h
# Create key vault enabled for deployment
Write-Host "Creating key vault:" $kvName
$kv = New-AzureRmKeyVault -ResourceGroupName $resourceGroup -VaultName $kvName -Location $location -Sku standard -EnabledForDeployment
Write-Host "=> Key Vault Resource ID:" $kv.ResourceId
# Serialize certificate
Write-Host "Setting access polices for client" $spnName
Set-AzureRmKeyVaultAccessPolicy -VaultName $kvName -ServicePrincipalName $spnName -PermissionsToSecrets GET,LIST
# Store certificate as secret
Write-Host "Storing certificate in key vault:" $pfxPath
$fileContentBytes = get-content $pfxPath -Encoding Byte
$fileContentEncoded = [System.Convert]::ToBase64String($fileContentBytes)
$jsonObject = @"
@ -74,59 +71,75 @@ $jsonObject = @"
$jsonObjectBytes = [System.Text.Encoding]::UTF8.GetBytes($jsonObject)
$jsonEncoded = [System.Convert]::ToBase64String($jsonObjectBytes)
$secret = ConvertTo-SecureString -String $jsonEncoded -AsPlainText -Force
# Upload certificate as secret
Write-Host "Storing certificate in key vault:" $pfxPath
$kvSecret = Set-AzureKeyVaultSecret -VaultName $kvName -Name $secretName -SecretValue $secret
$kvSecret = Set-AzureKeyVaultSecret -VaultName $kvName -Name $pfxSecret -SecretValue $secret -ContentType pfx
# Compute certificate thumbprint
Write-Host "Computing certificate thumbprint"
$tp = Get-PfxCertificate -FilePath $pfxPath
Write-Host "=> Certificate URL:" $kvSecret.Id
Write-Host "=> Certificate thumbprint:" $tp.Thumbprint
Write-Host "Storing secret for sample user: $userName"
$userSecret = ConvertTo-SecureString -String $userPass -AsPlainText -Force
Set-AzureKeyVaultSecret -VaultName $kvName -Name $userName -SecretValue $userSecret -ContentType "user credentials" | out-null
# BUILD TEMPLATE PARAMETERS JSON
# =============================================
$jsonParameters = New-Object -TypeName PSObject
$jsonStorageAccountName = New-Object -TypeName PSObject
$jsonStorageAccountName | Add-Member -MemberType NoteProperty -Name value -Value $saName
$jsonParameters | Add-Member -MemberType NoteProperty -Name storageAccountName -Value $jsonStorageAccountName
$jsonAdminPublicKey = New-Object -TypeName PSObject
$jsonAdminPublicKey | Add-Member -MemberType NoteProperty -Name value -Value $sshKey
$jsonParameters | Add-Member -MemberType NoteProperty -Name adminPublicKey -Value $jsonAdminPublicKey
$jsonVirtualMachineSize = New-Object -TypeName PSObject
$jsonVirtualMachineSize | Add-Member -MemberType NoteProperty -Name value -Value $vmSize
$jsonParameters | Add-Member -MemberType NoteProperty -Name virtualMachineSize -Value $jsonVirtualMachineSize
$jsonPipName = New-Object -TypeName PSObject
$jsonPipName | Add-Member -MemberType NoteProperty -Name value -Value $dnsLabelName
$jsonParameters | Add-Member -MemberType NoteProperty -Name pipName -Value $jsonPipName
$jsonPipDomainNameLabel = New-Object -TypeName PSObject
$jsonPipDomainNameLabel | Add-Member -MemberType NoteProperty -Name value -Value $dnsLabelName
$jsonParameters | Add-Member -MemberType NoteProperty -Name pipDomainNameLabel -Value $jsonPipDomainNameLabel
$jsonStorageAccountResourceId = New-Object -TypeName PSObject
$jsonStorageAccountResourceId | Add-Member -MemberType NoteProperty -Name value -Value $sa.Id
$jsonParameters | Add-Member -MemberType NoteProperty -Name storageAccountResourceId -Value $jsonStorageAccountResourceId
$jsonStorageAccountContainerName = New-Object -TypeName PSObject
$jsonStorageAccountContainerName | Add-Member -MemberType NoteProperty -Name value -Value $saContainer
$jsonParameters | Add-Member -MemberType NoteProperty -Name storageAccountContainer -Value $jsonStorageAccountContainerName
$jsonStorageAccountKey = New-Object -TypeName PSObject
$jsonStorageAccountKey | Add-Member -MemberType NoteProperty -Name value -Value $saKey
$jsonParameters | Add-Member -MemberType NoteProperty -Name storageAccountKey -Value $jsonStorageAccountKey
$jsonKeyVaultResourceId = New-Object -TypeName PSObject
$jsonKeyVaultResourceId | Add-Member -MemberType NoteProperty -Name value -Value $kv.ResourceId
$jsonParameters | Add-Member -MemberType NoteProperty -Name keyVaultResourceId -Value $jsonKeyVaultResourceId
$jsonParameters | Add-Member -MemberType NoteProperty -Name pfxKeyVaultResourceId -Value $jsonKeyVaultResourceId
$jsonKeyVaultSecretUrl = New-Object -TypeName PSObject
$jsonKeyVaultSecretUrl | Add-Member -MemberType NoteProperty -Name value -Value $kvSecret.Id
$jsonParameters | Add-Member -MemberType NoteProperty -Name keyVaultSecretUrl -Value $jsonKeyVaultSecretUrl
$jsonParameters | Add-Member -MemberType NoteProperty -Name pfxKeyVaultSecretUrl -Value $jsonKeyVaultSecretUrl
$jsonCertificateThumbprint = New-Object -TypeName PSObject
$jsonCertificateThumbprint | Add-Member -MemberType NoteProperty -Name value -Value $tp.Thumbprint
$jsonParameters | Add-Member -MemberType NoteProperty -Name certificateThumbprint -Value $jsonCertificateThumbprint
$jsonParameters | Add-Member -MemberType NoteProperty -Name pfxThumbprint -Value $jsonCertificateThumbprint
$jsonAdminPublicKey = New-Object -TypeName PSObject
$jsonAdminPublicKey | Add-Member -MemberType NoteProperty -Name value -Value $sshKey
$jsonParameters | Add-Member -MemberType NoteProperty -Name adminPublicKey -Value $jsonAdminPublicKey
$jsonRegistryTag = New-Object -TypeName PSObject
$jsonRegistryTag | Add-Member -MemberType NoteProperty -Name value -Value $registryTag
$jsonParameters | Add-Member -MemberType NoteProperty -Name registryTag -Value $jsonRegistryTag
$jsonDomainNameLabel = New-Object -TypeName PSObject
$jsonDomainNameLabel | Add-Member -MemberType NoteProperty -Name value -Value $dnsSubDomain
$jsonParameters | Add-Member -MemberType NoteProperty -Name domainNameLabel -Value $jsonDomainNameLabel
$jsonRegistryReplicas = New-Object -TypeName PSObject
$jsonRegistryReplicas | Add-Member -MemberType NoteProperty -Name value -Value $registryReplicas
$jsonParameters | Add-Member -MemberType NoteProperty -Name registryReplicas -Value $jsonRegistryReplicas
$jsonCseLocation = New-Object -TypeName PSObject
$jsonCseLocation | Add-Member -MemberType NoteProperty -Name value -Value $cseUrl
$jsonParameters | Add-Member -MemberType NoteProperty -Name cseLocation -Value $jsonCseLocation
$jsonSpnName = New-Object -TypeName PSObject
$jsonSpnName | Add-Member -MemberType NoteProperty -Name value -Value $spnName
$jsonParameters | Add-Member -MemberType NoteProperty -Name servicePrincipalClientId -Value $jsonSpnName
$jsonHtpasswdLocation = New-Object -TypeName PSObject
$jsonHtpasswdLocation | Add-Member -MemberType NoteProperty -Name value -Value $htpasswdUrl
$jsonParameters | Add-Member -MemberType NoteProperty -Name htpasswdLocation -Value $jsonHtpasswdLocation
$jsonSpnSecret = New-Object -TypeName PSObject
$jsonSpnSecret | Add-Member -MemberType NoteProperty -Name value -Value $spnSecret
$jsonParameters | Add-Member -MemberType NoteProperty -Name servicePrincipalClientSecret -Value $jsonSpnSecret
$jsonRoot = New-Object -TypeName PSObject
$jsonRoot | Add-Member -MemberType NoteProperty -Name schema -Value "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#"