Merge pull request #10 from neudesic/cloudera
Initial commit of cloudera_on_centos template
This commit is contained in:
Kay Singh
Коммит
87181c576b
|
|
@ -0,0 +1 @@
|
|||
*.sh -crlf
|
||||
|
|
@ -0,0 +1,114 @@
|
|||
|
||||
# Deploy a Cloudera CDH installation on CentOS virtual machines
|
||||
|
||||
<a href="https://azuredeploy.net/" target="_blank">
|
||||
<img src="http://azuredeploy.net/deploybutton.png" />
|
||||
</a>
|
||||
|
||||
This template creates a multi-server Cloudera CDH 5.3.3 Apache Hadoop deployment on CentOS virtual machines, and configures the CDH installation for a high availability cluster.
|
||||
The template also provisions storage accounts, virtual network, availability set, network interfaces, VMs, disks and other infrastructure and runtime resources required by the installation.
|
||||
|
||||
The template expects the following parameters:
|
||||
|
||||
| Name | Description | Default Value |
|
||||
|:--- |:---|:---|
|
||||
| adminUsername | Administrator user name used when provisioning virtual machines | testuser |
|
||||
| adminPassword | Administrator password used when provisioning virtual machines | Eur32#1e |
|
||||
| storageAccountPrefix | Unique namespace for the Storage Account where the Virtual Machine's disks will be placed | defaultStorageAccountPrefix |
|
||||
| dnsNamePrefix | Unique public dns name where the Virtual Machines will be exposed | defaultDnsNamePrefix |
|
||||
| region | Azure data center location where resources will be provisioned | |
|
||||
| storageAccountType | The type of the Storage Account to be created | Standard_LRS |
|
||||
| virtualNetworkName | The name of the virtual network provisioned for the deployment | clouderaVnet |
|
||||
| subnetName | Subnet name for the virtual network where resources will be provisioned | clouderaSubnet |
|
||||
| addressPrefix | The network address space for the virtual network | 10.0.0.0/24 |
|
||||
| subnetPrefix | The network address space for the virtual subnet | 10.0.0.0/24 |
|
||||
| nodeAddressPrefix | The IP address prefix that will be used for constructing private IP address for each node in the cluster | 10.0.0. |
|
||||
| tshirtSize | T-shirt size of the Cloudera cluster (Eval, Small, Medium, Large) | Eval |
|
||||
| vmSize | The size of the VMs deployed in the cluster (Defaults to Standard_D14) | Standard_D14 |
|
||||
| publicSSHCert | The base64 representation of the certificate used for SSH login. Details in Key Vault and SSH Keys section below. | Read section below |
|
||||
| keyVaultResourceGroup | The resource group containing the key vault which provides the private key used for SSH login. | AzureRM-Util |
|
||||
| keyVaultName | The name of the key vault which provides the private key used for SSH login. | AzureRM-Keys |
|
||||
| keyUri | The url of the private key used for SSH login. Details in Key Vault and SSH Keys section below. | Read section below |
|
||||
|
||||
|
||||
Topology
|
||||
--------
|
||||
|
||||
The deployment topology is comprised of a predefined number (as per t-shirt sizing) Cloudera member nodes configured as a cluster, configured using a set number of master,
|
||||
name and data nodes. Typical setup for Cloudera uses one master node and 2 name nodes with as many data nodes are needed for the size that has been choosen ranging from as
|
||||
few as 3 to thousands of data nodes. The current template will scale at the highest end to 200 data nodes when using the large t-shirt size.
|
||||
|
||||
The following table outlines the deployment topology characteristics for each supported t-shirt size:
|
||||
|
||||
| T-Shirt Size | Member Node VM Size | CPU Cores | Memory | Data Disks | # of Master Node VMs | # of Name Node VMs | # of Data Node VMs |
|
||||
|:--- |:---|:---|:---|:---|:---|:---|:---|
|
||||
| Eval | Standard_D14 | 16 | 112 GB | 16x1000 GB | 1 | 2 | 3 |
|
||||
| Small | Standard_D14 | 16 | 112 GB | 16x1000 GB | 1 | 2 | 9 |
|
||||
| Medium | Standard_D14 | 16 | 112 GB | 16x1000 GB | 1 | 2 | 50 |
|
||||
| Large | Standard_D14 | 16 | 112 GB | 16x1000 GB | 1 | 2 | 200 |
|
||||
|
||||
##Notes, Known Issues & Limitations
|
||||
- All nodes in the cluster have a public IP qaddress.
|
||||
- Using passwords via SSH are disabled. Private keys should be used to access the nodes in the cluster (See notes below.)
|
||||
- The deployment script is not yet idempotent and cannot handle updates (although it currently works for initial provisioning only)
|
||||
- SSH key is not yet implemented and the template currently takes a password for the admin user
|
||||
- If security is a concern, do not use the provided .pfx file
|
||||
|
||||
##Connecting to the cluster
|
||||
The machines are named according to a specific pattern. The master node is named based on parameters and using the.
|
||||
|
||||
[dnsNamePrefix]-mn.[region].cloudapp.azure.com
|
||||
|
||||
If the dnsNamePrefix was clouderatest in the West US region, the machine will be located at:
|
||||
|
||||
clouderatest-mn.westus.cloudapp.azure.com
|
||||
|
||||
The name nodes and data nodes of the cluster use the same pattern, but with -nn and -dn extensions followed by their number. For example:
|
||||
|
||||
clouderatest-nn0.westus.cloudapp.azure.com
|
||||
clouderatest-nn1.westus.cloudapp.azure.com
|
||||
clouderatest-dn0.westus.cloudapp.azure.com
|
||||
clouderatest-dn1.westus.cloudapp.azure.com
|
||||
clouderatest-dn2.westus.cloudapp.azure.com
|
||||
|
||||
To connect to the master node via SSH, use the .pem key in the repository if you used the provided key or your own .pem file. See the section below for more information on SSH keys.
|
||||
|
||||
ssh -i server-cert.pem testuser@clouderatest-mn.westus.cloudapp.azure.com
|
||||
|
||||
##Managing SSH Keys
|
||||
The Cloudera cluster uses SSH to communicate between machines during the provisioning process. A public/private key pair is used to provide authentication between the machines and must be provided at provisioning time. A sample .pfx file is included and some steps must be taken to prepare it for use:
|
||||
- The pfx file must be uploaded to a key vault
|
||||
- The certificate must be extracted and provided as a parameter to the deployment
|
||||
- The private key can be extracted and used to connect to the cluster via SSH
|
||||
|
||||
###Uploading the .pfx to the Key Vault
|
||||
Creating the Key Vault and uploading the .pfx is done using a set of PowerShell scripts available [here](https://gallery.technet.microsoft.com/scriptcenter/Azure-Key-Vault-Powershell-1349b091). Download these scripts and load them into a PowerShell instance using the following script.
|
||||
|
||||
import-module .\KeyVaultManager
|
||||
|
||||
Now execute the [upload-keys.ps1](upload-keys.ps1) script found in this repository with the following parameters.
|
||||
|
||||
# resourceGroupName - the name of the resource group that will hold the key
|
||||
# region - Must be the same resource the cluster will be on
|
||||
# keyVaultName - A unique key vault name between 3-24 alpha-numeric characters
|
||||
# keyName - The name used to identify the key
|
||||
# pfxFile - The pfx file containing the certificate and private key
|
||||
|
||||
.\upload-keys.ps1 "TestKeyGroup" "East Asia" "predeekc123221TestKeyVault" "TestKey"
|
||||
.\server-cert.pfx
|
||||
|
||||
The output of the script will contain a URL that is used for the **keyUri** parameter. The rest of the the **resourceGroupName** and **keyVaultName** used in the script above will be used for the **keyVaultResourceGroup** and **keyVaultName**.
|
||||
|
||||
###Extracting the certificate from the pfx file
|
||||
OpenSSL will extract a certificate from a .pfx and print it to the console.
|
||||
|
||||
# The sample server-cert.pfx has no password
|
||||
openssl pkcs12 -in server-cert.pfx -clcerts -nokeys
|
||||
|
||||
The output of this command contains a base64 encoded string representing the certificate. This string as a single line between the BEGIN CERTIFICATE and END CERTIFICATE delimiters is used for the **publicSSHCert** parameter. If you use the provided server-cert.pfx file, the default parameter for **publicSSHCert** will work correctly.
|
||||
|
||||
###Extracting the private key from the pfx file
|
||||
OpenSSL will also extract the private key that can be used when connecting to the machine via SSL.
|
||||
|
||||
openssl pkcs12 -in server-cert.pfx -nocerts | openssl rsa -out server-cert.pem
|
||||
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
{
|
||||
"dnsNamePrefix": {
|
||||
"value": "defaultDnsNamePrefix"
|
||||
},
|
||||
"storageAccountPrefix": {
|
||||
"value": "defaultStorageAccountPrefix"
|
||||
},
|
||||
"region": {
|
||||
"value": "East Asia"
|
||||
},
|
||||
"tshirtSize": {
|
||||
"value": "Eval"
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,306 @@
|
|||
{
|
||||
"$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json",
|
||||
"contentVersion": "1.0.0.0",
|
||||
"parameters" : {
|
||||
"adminUserName": {
|
||||
"type": "string",
|
||||
"defaultValue": "testuser",
|
||||
"metadata": {
|
||||
"Description": "Admin user name for the Virtual Machines"
|
||||
}
|
||||
},
|
||||
"adminPassword": {
|
||||
"type": "securestring",
|
||||
"defaultValue": "Eur32#1e",
|
||||
"metadata": {
|
||||
"Description": "Admin password for the Virtual Machines"
|
||||
}
|
||||
},
|
||||
"storageAccountPrefix": {
|
||||
"type": "string",
|
||||
"defaultValue": "defaultStorageAccountPrefix",
|
||||
"metadata": {
|
||||
"Description": "Unique namespace for the Storage Account where the Virtual Machine's disks will be placed"
|
||||
}
|
||||
},
|
||||
"dnsNamePrefix": {
|
||||
"type": "string",
|
||||
"defaultValue": "defaultDnsNamePrefix",
|
||||
"metadata": {
|
||||
"Description": "Unique public dns name where the Virtual Machines will be exposed"
|
||||
}
|
||||
},
|
||||
"region": {
|
||||
"type": "string",
|
||||
"metadata": {
|
||||
"Description": "Location where resources will be provisioned"
|
||||
}
|
||||
},
|
||||
"storageAccountType": {
|
||||
"type": "string",
|
||||
"defaultValue": "Standard_LRS",
|
||||
"metadata": {
|
||||
"Description": "The type of the Storage Account to be created (Default to Standard_LRS)"
|
||||
}
|
||||
},
|
||||
"virtualNetworkName": {
|
||||
"type": "string",
|
||||
"defaultValue": "clouderaVnet",
|
||||
"metadata": {
|
||||
"Description": "The name of the virtual network provisioned for the deployment"
|
||||
}
|
||||
},
|
||||
"subnetName": {
|
||||
"type": "string",
|
||||
"defaultValue": "clouderaSubnet",
|
||||
"metadata": {
|
||||
"Description": "Subnet name for the virtual network where resources will be provisioned"
|
||||
}
|
||||
},
|
||||
"addressPrefix": {
|
||||
"type": "string",
|
||||
"defaultValue": "10.0.0.0/24",
|
||||
"metadata": {
|
||||
"Description": "The network address space for the virtual network"
|
||||
}
|
||||
},
|
||||
"subnetPrefix": {
|
||||
"type": "string",
|
||||
"defaultValue": "10.0.0.0/24",
|
||||
"metadata": {
|
||||
"Description": "The network address space for the virtual subnet"
|
||||
}
|
||||
},
|
||||
"nodeAddressPrefix": {
|
||||
"type": "string",
|
||||
"defaultValue": "10.0.0.",
|
||||
"metadata": {
|
||||
"Description": "The IP address prefix that will be used for constructing private IP address for each node in the cluster"
|
||||
}
|
||||
},
|
||||
"tshirtSize": {
|
||||
"type": "string",
|
||||
"defaultValue": "Eval",
|
||||
"allowedValues": [
|
||||
"Eval",
|
||||
"Small",
|
||||
"Medium"
|
||||
],
|
||||
"metadata": {
|
||||
"Description": "T-shirt size of the Cloudera cluster (Eval, Small, Medium)"
|
||||
}
|
||||
},
|
||||
"vmSize": {
|
||||
"type": "string",
|
||||
"defaultValue": "Standard_D14",
|
||||
"metadata": {
|
||||
"Description": "The size of the VMs deployed in the cluster (Defaults to Standard_D14)"
|
||||
}
|
||||
},
|
||||
"publicSSHCert":{
|
||||
"type" : "string",
|
||||
"defaultValue":"MIIDXTCCAkWgAwIBAgIJALKk5OArjHCZMA0GCSqGSIb3DQEBCwUAMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwHhcNMTUwNDE1MTY0NjU4WhcNMTYwNDE0MTY0NjU4WjBFMQswCQYDVQQGEwJBVTETMBEGA1UECAwKU29tZS1TdGF0ZTEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuM9aF13nZ+HIhP76XtX8Oa5p5whhZ0df6Rmho/fIwe1MB7iuMD7YjbZmIMunB6vx7oou0I7oAh9O79THODOo4jBrFEtfclmG/kEjjXliE7zAEeMi84caSEv4fcWbj5Sf591i6wcakQzqRVi6bMhuRlUFMQPT9XkcGkR9mOrLgHIKl9yRsahp1yBkhWBCcdVJB2BI4Tmb5keIbAhQsk9DSqWG2VTlH2vCC80oQDcHgJvplsWK88LCBO1qsuS9+WJjGcl4ADgQtZeKa8Uo9obdC7sncBZHOCvTXHvmdtCOiV4ZIDwUwEpAaPsA7ppqt4/FpuAizVAdiD3Ln8HfwdFahwIDAQABo1AwTjAdBgNVHQ4EFgQUaYkP2ppru53M3PAtW7Yzgms5KZMwHwYDVR0jBBgwFoAUaYkP2ppru53M3PAtW7Yzgms5KZMwDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAtpgkv4IAyVYUiS/4BXCgge+bnVphSHQVJgBki1v8FmyfTye70lVlquDYNi+cife+qgdGrFA46TwAn3B76RgW++gPjMI2drm0PcZ+EDWGqYmGhJkKL35T9j5dmd68qPLd8k5U7IE9Wr21JbQbsTB3O37geQ1jutaZD7dhwR+9FhQd4dyIus4ayBe/Z+IsrKmq5Iff9hTI71NfQbWSoeSePYretopxJt2XAUdmAWWI0U/X37etajLL8tONLJi8TSTQs75LFweoxmyfA7J8j4Oni6DxZwuKtfPL4HAdAcpRO5XdThP2WaPdLfCz3asSCE2K5IJ7rXfmiQ9oAptqNlJfmg==",
|
||||
"metadata": {
|
||||
"Description": "The base64 representation of the certificate used for SSH login. See the readme for details."
|
||||
}
|
||||
},
|
||||
"keyVaultResourceGroup": {
|
||||
"type": "string",
|
||||
"defaultValue": "AzureRM-Util",
|
||||
"metadata": {
|
||||
"Description": "The resource group containing the key vault which provides the private key used for SSH login. See the readme for details."
|
||||
}
|
||||
},
|
||||
"keyVaultName": {
|
||||
"type": "string",
|
||||
"defaultValue": "AzureRM-Keys",
|
||||
"metadata": {
|
||||
"Description": "The name of the key vault which provides the private key used for SSH login. See the readme for details."
|
||||
}
|
||||
},
|
||||
"keyUri": {
|
||||
"type": "string",
|
||||
"defaultValue": "https://azurerm-keys.vault.azure.net/secrets/sshKey/29a10ecf43b64c7ab7eace8866d64597",
|
||||
"metadata": {
|
||||
"Description": "The url of the private key used for SSH login. See the readme for details."
|
||||
}
|
||||
}
|
||||
},
|
||||
"variables": {
|
||||
"scriptsUri": "https://raw.githubusercontent.com/neudesic/azure-quickstart-templates/cloudera/cloudera-on-centos",
|
||||
"storageAccount": {
|
||||
"prefix": "[parameters('storageAccountPrefix')]",
|
||||
"type": "[parameters('storageAccountType')]"
|
||||
},
|
||||
"clusterTshirtSizeEval": {
|
||||
"nameNodeCount": 2,
|
||||
"dataNodeCount": 3
|
||||
},
|
||||
"clusterTshirtSizeSmall": {
|
||||
"nameNodeCount": 2,
|
||||
"dataNodeCount": 7
|
||||
},
|
||||
"clusterTshirtSizeMedium": {
|
||||
"nameNodeCount": 2,
|
||||
"dataNodeCount": 50
|
||||
},
|
||||
"clusterSpec": "[variables(concat('clusterTshirtSize', parameters('tshirtSize')))]",
|
||||
"vmSpec": {
|
||||
"vmSize": "[parameters('vmSize')]",
|
||||
"privateIPAddressPrefix": "[parameters('nodeAddressPrefix')]",
|
||||
"adminUserName": "[parameters('adminUserName')]",
|
||||
"adminPassword": "[parameters('adminPassword')]",
|
||||
"sourceImageName": "[concat('/',subscription().subscriptionId,'/services/images/5112500ae3b842c8b9c604889f8753c3__OpenLogic-CentOS-65-20150128')]",
|
||||
"keyVaultResourceId": "[resourceId(parameters('keyVaultResourceGroup'), 'Microsoft.KeyVault/vaults', parameters('keyVaultName'))]",
|
||||
"keyUri": "[parameters('keyUri')]",
|
||||
"publicSSHCert": "[parameters('publicSSHCert')]"
|
||||
},
|
||||
"networkSpec": {
|
||||
"addressPrefix": "[parameters('addressPrefix')]",
|
||||
"subnetPrefix": "[parameters('subnetPrefix')]",
|
||||
"privateIPPrefix": "[parameters('nodeAddressPrefix')]",
|
||||
"masterNodeAddress": 9,
|
||||
"nameNodeAddressStart": 10,
|
||||
"dataNodeAddressStart": 20,
|
||||
"virtualNetworkName": "[parameters('virtualNetworkName')]",
|
||||
"virtualNetworkSubnetName": "[parameters('subnetName')]"
|
||||
}
|
||||
},
|
||||
"resources": [
|
||||
{
|
||||
"name": "shared-resources",
|
||||
"type": "Microsoft.Resources/deployments",
|
||||
"apiVersion": "2015-01-01",
|
||||
"properties": {
|
||||
"mode": "Incremental",
|
||||
"templateLink": {
|
||||
"uri": "[concat(variables('scriptsUri'), '/shared-resources.json')]",
|
||||
"contentVersion": "1.0.0.0"
|
||||
},
|
||||
"parameters": {
|
||||
"region": {
|
||||
"value": "[parameters('region')]"
|
||||
},
|
||||
"networkSpec": {
|
||||
"value": "[variables('networkSpec')]"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "name-node",
|
||||
"type": "Microsoft.Resources/deployments",
|
||||
"apiVersion": "2015-01-01",
|
||||
"dependsOn": [
|
||||
"Microsoft.Resources/deployments/shared-resources"
|
||||
],
|
||||
"properties": {
|
||||
"mode": "Incremental",
|
||||
"templateLink": {
|
||||
"uri": "[concat(variables('scriptsUri'), '/name-node.json')]",
|
||||
"contentVersion": "1.0.0.0"
|
||||
},
|
||||
"parameters": {
|
||||
"dnsNamePrefix": {
|
||||
"value": "[parameters('dnsNamePrefix')]"
|
||||
},
|
||||
"region": {
|
||||
"value": "[parameters('region')]"
|
||||
},
|
||||
"scriptsUri": {
|
||||
"value": "[variables('scriptsUri')]"
|
||||
},
|
||||
"storageAccount" : {
|
||||
"value": "[variables('storageAccount')]"
|
||||
},
|
||||
"vmCount": {
|
||||
"value": "[variables('clusterSpec').nameNodeCount]"
|
||||
},
|
||||
"vmSpec": {
|
||||
"value": "[variables('vmSpec')]"
|
||||
},
|
||||
"networkSpec": {
|
||||
"value": "[variables('networkSpec')]"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "data-node",
|
||||
"type": "Microsoft.Resources/deployments",
|
||||
"apiVersion": "2015-01-01",
|
||||
"dependsOn": [
|
||||
"Microsoft.Resources/deployments/shared-resources"
|
||||
],
|
||||
"properties": {
|
||||
"mode": "Incremental",
|
||||
"templateLink": {
|
||||
"uri": "[concat(variables('scriptsUri'), '/data-node.json')]",
|
||||
"contentVersion": "1.0.0.0"
|
||||
},
|
||||
"parameters": {
|
||||
"dnsNamePrefix": {
|
||||
"value": "[parameters('dnsNamePrefix')]"
|
||||
},
|
||||
"region": {
|
||||
"value": "[parameters('region')]"
|
||||
},
|
||||
"scriptsUri": {
|
||||
"value": "[variables('scriptsUri')]"
|
||||
},
|
||||
"storageAccount" : {
|
||||
"value": "[variables('storageAccount')]"
|
||||
},
|
||||
"vmCount": {
|
||||
"value": "[variables('clusterSpec').dataNodeCount]"
|
||||
},
|
||||
"vmSpec": {
|
||||
"value": "[variables('vmSpec')]"
|
||||
},
|
||||
"networkSpec": {
|
||||
"value": "[variables('networkSpec')]"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "master-node",
|
||||
"type": "Microsoft.Resources/deployments",
|
||||
"apiVersion": "2015-01-01",
|
||||
"dependsOn": [
|
||||
"Microsoft.Resources/deployments/data-node",
|
||||
"Microsoft.Resources/deployments/name-node"
|
||||
],
|
||||
"properties": {
|
||||
"mode": "Incremental",
|
||||
"templateLink": {
|
||||
"uri": "[concat(variables('scriptsUri'), '/master-node.json')]",
|
||||
"contentVersion": "1.0.0.0"
|
||||
},
|
||||
"parameters": {
|
||||
"dnsNamePrefix": {
|
||||
"value": "[parameters('dnsNamePrefix')]"
|
||||
},
|
||||
"region": {
|
||||
"value": "[parameters('region')]"
|
||||
},
|
||||
"scriptsUri": {
|
||||
"value": "[variables('scriptsUri')]"
|
||||
},
|
||||
"storageAccount" : {
|
||||
"value": "[variables('storageAccount')]"
|
||||
},
|
||||
"vmSpec": {
|
||||
"value": "[variables('vmSpec')]"
|
||||
},
|
||||
"networkSpec" : {
|
||||
"value": "[variables('networkSpec')]"
|
||||
},
|
||||
"clusterSpec" : {
|
||||
"value": "[variables('clusterSpec')]"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,310 @@
|
|||
{
|
||||
"$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
|
||||
"contentVersion": "1.0.0.0",
|
||||
"parameters": {
|
||||
"dnsNamePrefix": {
|
||||
"type": "string"
|
||||
},
|
||||
"region": {
|
||||
"type": "string"
|
||||
},
|
||||
"scriptsUri": {
|
||||
"type": "string"
|
||||
},
|
||||
"storageAccount" : {
|
||||
"type": "object"
|
||||
},
|
||||
"vmCount": {
|
||||
"type":"int"
|
||||
},
|
||||
"vmSpec": {
|
||||
"type": "object"
|
||||
},
|
||||
"networkSpec": {
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"variables": {
|
||||
"vmName": "[concat(parameters('dnsNamePrefix'), '-dn')]",
|
||||
"nicName": "[concat(variables('vmName'), '-nic')]",
|
||||
"storageAccountName": "[concat(parameters('storageAccount').prefix, 'dn')]",
|
||||
|
||||
"publicIPAddressName": "[concat(variables('vmName'), '-publicIP')]",
|
||||
},
|
||||
"resources": [
|
||||
{
|
||||
"type": "Microsoft.Storage/storageAccounts",
|
||||
"name": "[concat(variables('storageAccountName'), copyIndex())]",
|
||||
"apiVersion": "2014-12-01-preview",
|
||||
"location": "[parameters('region')]",
|
||||
"copy": {
|
||||
"name": "storageAccountLoop",
|
||||
"count": "[parameters('vmCount')]"
|
||||
},
|
||||
"properties": {
|
||||
"accountType": "[parameters('storageAccount').type]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"apiVersion": "2014-12-01-preview",
|
||||
"type": "Microsoft.Network/publicIPAddresses",
|
||||
"name": "[concat(variables('publicIPAddressName'), copyIndex())]",
|
||||
"location": "[parameters('region')]",
|
||||
"copy": {
|
||||
"name": "publicIPLoop",
|
||||
"count": "[parameters('vmCount')]"
|
||||
},
|
||||
"properties": {
|
||||
"publicIPAllocationMethod": "Dynamic",
|
||||
"dnsSettings": {
|
||||
"domainNameLabel": "[concat(parameters('dnsNamePrefix'), '-dn', copyIndex())]"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"apiVersion": "2014-12-01-preview",
|
||||
"type": "Microsoft.Network/networkInterfaces",
|
||||
"name": "[concat(variables('nicName'), copyIndex())]",
|
||||
"location": "[parameters('region')]",
|
||||
"dependsOn": [
|
||||
"[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'), copyIndex())]",
|
||||
"[concat('Microsoft.Storage/storageAccounts/', variables('storageAccountName'), copyIndex())]"
|
||||
],
|
||||
"copy": {
|
||||
"name": "nameNodeNicLoop",
|
||||
"count": "[parameters('vmCount')]"
|
||||
},
|
||||
"properties": {
|
||||
"ipConfigurations": [{
|
||||
"name": "ipconfig1",
|
||||
"properties": {
|
||||
"privateIPAllocationMethod": "Static",
|
||||
"privateIPAddress": "[concat(parameters('networkSpec').privateIPPrefix, copyIndex(parameters('networkSpec').dataNodeAddressStart))]",
|
||||
"subnet": {
|
||||
"id": "[concat(resourceId('Microsoft.Network/virtualNetworks',parameters('networkSpec').virtualNetworkName), '/subnets/', parameters('networkSpec').virtualNetworkSubnetName)]"
|
||||
},
|
||||
"publicIPAddress": {
|
||||
"id": "[resourceId('Microsoft.Network/publicIPAddresses',concat(variables('publicIPAddressName'), copyIndex()))]"
|
||||
}
|
||||
}
|
||||
}]
|
||||
}
|
||||
},
|
||||
{
|
||||
"apiVersion": "2014-12-01-preview",
|
||||
"type": "Microsoft.Compute/virtualMachines",
|
||||
"name": "[concat(variables('vmName'), copyIndex())]",
|
||||
"location": "[parameters('region')]",
|
||||
"copy": {
|
||||
"name": "nameNodeNicLoop",
|
||||
"count": "[parameters('vmCount')]"
|
||||
},
|
||||
"dependsOn": [
|
||||
"[concat('Microsoft.Network/networkInterfaces/', variables('nicName'), copyIndex())]"
|
||||
],
|
||||
"properties": {
|
||||
"hardwareProfile": {
|
||||
"vmSize": "[parameters('vmSpec').vmSize]"
|
||||
},
|
||||
"osProfile": {
|
||||
"computername": "[concat(variables('vmName'), copyIndex())]",
|
||||
"adminUsername": "[parameters('vmSpec').adminUserName]",
|
||||
"adminPassword": "[parameters('vmSpec').adminPassword]",
|
||||
"linuxConfiguration" : {
|
||||
"disablePasswordAuthentication": "true",
|
||||
"ssh": {
|
||||
"publicKeys": [
|
||||
{
|
||||
"keyData": "[parameters('vmSpec').publicSSHCert]",
|
||||
"path": "[concat('/home/',parameters('vmSpec').adminUserName,'/.ssh/authorized_keys')]"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"storageProfile": {
|
||||
"sourceImage": {
|
||||
"id": "[parameters('vmSpec').sourceImageName]"
|
||||
},
|
||||
"destinationVhdsContainer" : "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/')]",
|
||||
"dataDisks": [
|
||||
{
|
||||
"name": "datadisk0",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 0,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk0.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk1",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 1,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk1.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk2",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 2,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk2.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk3",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 3,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk3.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk4",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 4,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk4.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk5",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 5,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk5.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk6",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 6,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk6.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk7",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 7,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk7.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk8",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 8,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk8.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk9",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 9,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk9.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk10",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 10,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk10.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk11",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 11,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk11.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk12",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 12,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk12.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk13",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 13,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk13.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk14",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 14,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk14.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk15",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 15,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk15.vhd')]"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"networkProfile": {
|
||||
"networkInterfaces" : [{
|
||||
"id": "[resourceId('Microsoft.Network/networkInterfaces',concat(variables('nicName'), copyIndex()))]"
|
||||
}],
|
||||
"inputEndpoints": [
|
||||
{
|
||||
"enableDirectServerReturn": "False",
|
||||
"endpointName": "HDFS.dn.Web.UI",
|
||||
"privatePort": 50075,
|
||||
"publicPort": 50075,
|
||||
"protocol": "tcp"
|
||||
},
|
||||
{
|
||||
"enableDirectServerReturn": "False",
|
||||
"endpointName": "YARN.nm.Web.UI",
|
||||
"privatePort": 8042,
|
||||
"publicPort": 8042,
|
||||
"protocol": "tcp"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.Compute/virtualMachines/extensions",
|
||||
"name": "[concat(variables('vmName'), copyIndex(), '/prepareDisks')]",
|
||||
"apiVersion": "2014-12-01-preview",
|
||||
"location": "[parameters('region')]",
|
||||
"copy": {
|
||||
"name": "nameNodeNicLoop",
|
||||
"count": "[parameters('vmCount')]"
|
||||
},
|
||||
"dependsOn": [
|
||||
"[concat('Microsoft.Compute/virtualMachines/', variables('vmName'), copyIndex())]"
|
||||
],
|
||||
"properties": {
|
||||
"publisher": "Microsoft.OSTCExtensions",
|
||||
"type": "CustomScriptForLinux",
|
||||
"typeHandlerVersion": "1.2",
|
||||
"settings": {
|
||||
"fileUris": [
|
||||
"[concat(parameters('scriptsUri'), '/scripts/prepareDisks.sh')]",
|
||||
"[concat(parameters('scriptsUri'), '/scripts/initialize-node.sh')]"
|
||||
],
|
||||
"commandToExecute": "sh initialize-node.sh"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,308 @@
|
|||
{
|
||||
"$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
|
||||
"contentVersion": "1.0.0.0",
|
||||
"parameters": {
|
||||
"dnsNamePrefix": {
|
||||
"type": "string"
|
||||
},
|
||||
"region": {
|
||||
"type": "string"
|
||||
},
|
||||
"scriptsUri": {
|
||||
"type": "string"
|
||||
},
|
||||
"storageAccount" : {
|
||||
"type": "object"
|
||||
},
|
||||
"vmSpec": {
|
||||
"type": "object"
|
||||
},
|
||||
"networkSpec": {
|
||||
"type": "object"
|
||||
},
|
||||
"clusterSpec": {
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"variables": {
|
||||
"vmName": "[concat(parameters('dnsNamePrefix'), '-mn')]",
|
||||
"publicIPAddressName": "[concat(variables('vmName'), '-publicIP')]",
|
||||
"nicName": "[concat(variables('vmName'), '-nic')]",
|
||||
"storageAccountName": "[concat(parameters('storageAccount').prefix, 'mn')]"
|
||||
},
|
||||
"resources": [
|
||||
{
|
||||
"type": "Microsoft.Storage/storageAccounts",
|
||||
"name": "[variables('storageAccountName')]",
|
||||
"apiVersion": "2014-12-01-preview",
|
||||
"location": "[parameters('region')]",
|
||||
"properties": {
|
||||
"accountType": "[parameters('storageAccount').type]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"apiVersion": "2014-12-01-preview",
|
||||
"type": "Microsoft.Network/publicIPAddresses",
|
||||
"name": "[variables('publicIPAddressName')]",
|
||||
"location": "[parameters('region')]",
|
||||
"properties": {
|
||||
"publicIPAllocationMethod": "Dynamic",
|
||||
"dnsSettings": {
|
||||
"domainNameLabel": "[concat(parameters('dnsNamePrefix'), '-mn')]"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"apiVersion": "2014-12-01-preview",
|
||||
"type": "Microsoft.Network/networkInterfaces",
|
||||
"name": "[variables('nicName')]",
|
||||
"location": "[parameters('region')]",
|
||||
"dependsOn": [
|
||||
"[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'))]"
|
||||
],
|
||||
"properties": {
|
||||
"ipConfigurations": [{
|
||||
"name": "ipconfig1",
|
||||
"properties": {
|
||||
"privateIPAllocationMethod": "Static",
|
||||
"privateIPAddress": "[concat(parameters('networkSpec').privateIPPrefix, parameters('networkSpec').masterNodeAddress)]",
|
||||
"publicIPAddress": {
|
||||
"id": "[resourceId('Microsoft.Network/publicIPAddresses',variables('publicIPAddressName'))]"
|
||||
},
|
||||
"subnet": {
|
||||
"id": "[concat(resourceId('Microsoft.Network/virtualNetworks',parameters('networkSpec').virtualNetworkName), '/subnets/', parameters('networkSpec').virtualNetworkSubnetName)]"
|
||||
}
|
||||
}
|
||||
}]
|
||||
}
|
||||
},
|
||||
{
|
||||
"apiVersion": "2014-12-01-preview",
|
||||
"type": "Microsoft.Compute/virtualMachines",
|
||||
"name": "[variables('vmName')]",
|
||||
"location": "[parameters('region')]",
|
||||
"dependsOn": [
|
||||
"[concat('Microsoft.Network/networkInterfaces/', variables('nicName'))]",
|
||||
"[concat('Microsoft.Storage/storageAccounts/', variables('storageAccountName'))]"
|
||||
],
|
||||
"properties": {
|
||||
"hardwareProfile": {
|
||||
"vmSize": "[parameters('vmSpec').vmSize]"
|
||||
},
|
||||
"osProfile": {
|
||||
"computername": "[variables('vmName')]",
|
||||
"adminUsername": "[parameters('vmSpec').adminUserName]",
|
||||
"adminPassword": "[parameters('vmSpec').adminPassword]",
|
||||
"secrets": [{
|
||||
"sourceVault": {
|
||||
"id": "[parameters('vmSpec').keyVaultResourceId]"
|
||||
},
|
||||
"vaultCertificates": [{
|
||||
"certificateUrl": "[parameters('vmSpec').keyUri]"
|
||||
}]
|
||||
}],
|
||||
"linuxConfiguration" : {
|
||||
"disablePasswordAuthentication": "true",
|
||||
"ssh": {
|
||||
"publicKeys": [
|
||||
{
|
||||
"keyData": "[parameters('vmSpec').publicSSHCert]",
|
||||
"path": "[concat('/home/',parameters('vmSpec').adminUserName,'/.ssh/authorized_keys')]"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"storageProfile": {
|
||||
"sourceImage": {
|
||||
"id": "[parameters('vmSpec').sourceImageName]"
|
||||
},
|
||||
"destinationVhdsContainer" : "[concat('http://',variables('storageAccountName'),'.blob.core.windows.net/vhds/')]",
|
||||
"dataDisks": [
|
||||
{
|
||||
"name": "datadisk0",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 0,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk0.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk1",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 1,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk1.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk2",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 2,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk2.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk3",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 3,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk3.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk4",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 4,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk4.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk5",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 5,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk5.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk6",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 6,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk6.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk7",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 7,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk7.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk8",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 8,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk8.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk9",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 9,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk9.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk10",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 10,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk10.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk11",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 11,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk11.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk12",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 12,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk12.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk13",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 13,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk13.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk14",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 14,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk14.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk15",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 15,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk15.vhd')]"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"networkProfile": {
|
||||
"networkInterfaces" : [{
|
||||
"id": "[resourceId('Microsoft.Network/networkInterfaces',variables('nicName'))]"
|
||||
}],
|
||||
"inputEndpoints": [
|
||||
{
|
||||
"enableDirectServerReturn": "False",
|
||||
"endpointName": "Management.Web.UI",
|
||||
"privatePort": 7180,
|
||||
"publicPort": 7180,
|
||||
"protocol": "tcp"
|
||||
},
|
||||
{
|
||||
"enableDirectServerReturn": "False",
|
||||
"endpointName": "Navigator",
|
||||
"privatePort": 7187,
|
||||
"publicPort": 7187,
|
||||
"protocol": "tcp"
|
||||
},
|
||||
{
|
||||
"enableDirectServerReturn": "False",
|
||||
"endpointName": "SSH",
|
||||
"privatePort": 22,
|
||||
"publicPort": 5222,
|
||||
"protocol": "tcp"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.Compute/virtualMachines/extensions",
|
||||
"name": "[concat(parameters('dnsNamePrefix'), '-mn', '/prepareDisks')]",
|
||||
"apiVersion": "2014-12-01-preview",
|
||||
"location": "[parameters('region')]",
|
||||
"dependsOn": [
|
||||
"[concat('Microsoft.Compute/virtualMachines/', variables('vmName'))]"
|
||||
],
|
||||
"properties": {
|
||||
"publisher": "Microsoft.OSTCExtensions",
|
||||
"type": "CustomScriptForLinux",
|
||||
"typeHandlerVersion": "1.2",
|
||||
"settings": {
|
||||
"fileUris": [
|
||||
"[concat(parameters('scriptsUri'), '/scripts/initialize-master.sh')]",
|
||||
"[concat(parameters('scriptsUri'), '/scripts/initialize-node.sh')]",
|
||||
"[concat(parameters('scriptsUri'), '/scripts/prepareDisks.sh')]",
|
||||
"[concat(parameters('scriptsUri'), '/scripts/bootstrap-cloudera.sh')]",
|
||||
"[concat(parameters('scriptsUri'), '/scripts/initialize-cloudera-server.sh')]",
|
||||
"[concat(parameters('scriptsUri'), '/scripts/cmxDeployOnIbiza.py')]"
|
||||
],
|
||||
"commandToExecute": "[concat('sh initialize-master.sh \"', parameters('networkSpec').privateIPPrefix, '\" \"', parameters('dnsNamePrefix'), '\" \"', reference(variables('publicIPAddressName')).dnsSettings.fqdn, '\" ', parameters('clusterSpec').nameNodeCount, ' ', parameters('clusterSpec').dataNodeCount, ' \"', parameters('vmSpec').adminUserName, '\"')]"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
"itemDisplayName": "Cloudera on CentOS VMs",
|
||||
"description": "This template creates a multi-server Cloudera deployment on CentOR virtual machines, and configures the Cloudera installation",
|
||||
"summary": "This template creates a multi-server Cloudera deployment on CentOS virtual machines",
|
||||
"githubUsername": "predeekc",
|
||||
"dateUpdated": "2015-04-18"
|
||||
}
|
||||
|
|
@ -0,0 +1,293 @@
|
|||
{
|
||||
"$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
|
||||
"contentVersion": "1.0.0.0",
|
||||
"parameters": {
|
||||
"dnsNamePrefix": {
|
||||
"type": "string"
|
||||
},
|
||||
"region": {
|
||||
"type": "string"
|
||||
},
|
||||
"scriptsUri": {
|
||||
"type": "string"
|
||||
},
|
||||
"storageAccount" : {
|
||||
"type": "object"
|
||||
},
|
||||
"vmCount": {
|
||||
"type":"int"
|
||||
},
|
||||
"vmSpec": {
|
||||
"type": "object"
|
||||
},
|
||||
"networkSpec": {
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"variables": {
|
||||
"vmName": "[concat(parameters('dnsNamePrefix'), '-nn')]",
|
||||
"nicName": "[concat(variables('vmName'), '-nic')]",
|
||||
"publicIPAddressName": "[concat(variables('vmName'), '-publicIP')]",
|
||||
"storageAccountName": "[concat(parameters('storageAccount').prefix, 'nn')]"
|
||||
},
|
||||
"resources": [
|
||||
{
|
||||
"type": "Microsoft.Storage/storageAccounts",
|
||||
"name": "[concat(variables('storageAccountName'), copyIndex())]",
|
||||
"apiVersion": "2014-12-01-preview",
|
||||
"location": "[parameters('region')]",
|
||||
"copy": {
|
||||
"name": "storageAccountLoop",
|
||||
"count": "[parameters('vmCount')]"
|
||||
},
|
||||
"properties": {
|
||||
"accountType": "[parameters('storageAccount').type]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"apiVersion": "2014-12-01-preview",
|
||||
"type": "Microsoft.Network/publicIPAddresses",
|
||||
"name": "[concat(variables('publicIPAddressName'), copyIndex())]",
|
||||
"location": "[parameters('region')]",
|
||||
"copy": {
|
||||
"name": "publicIPLoop",
|
||||
"count": "[parameters('vmCount')]"
|
||||
},
|
||||
"properties": {
|
||||
"publicIPAllocationMethod": "Dynamic",
|
||||
"dnsSettings": {
|
||||
"domainNameLabel": "[concat(parameters('dnsNamePrefix'), '-nn', copyIndex())]"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"apiVersion": "2014-12-01-preview",
|
||||
"type": "Microsoft.Network/networkInterfaces",
|
||||
"name": "[concat(variables('nicName'), copyIndex())]",
|
||||
"location": "[parameters('region')]",
|
||||
"dependsOn": [
|
||||
"[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'), copyIndex())]"
|
||||
],
|
||||
"copy": {
|
||||
"name": "nameNodeNicLoop",
|
||||
"count": "[parameters('vmCount')]"
|
||||
},
|
||||
"properties": {
|
||||
"ipConfigurations": [{
|
||||
"name": "ipconfig1",
|
||||
"properties": {
|
||||
"privateIPAllocationMethod": "Static",
|
||||
"privateIPAddress": "[concat(parameters('networkSpec').privateIPPrefix, copyIndex(parameters('networkSpec').nameNodeAddressStart))]",
|
||||
"subnet": {
|
||||
"id": "[concat(resourceId('Microsoft.Network/virtualNetworks',parameters('networkSpec').virtualNetworkName), '/subnets/', parameters('networkSpec').virtualNetworkSubnetName)]"
|
||||
},
|
||||
"publicIPAddress": {
|
||||
"id": "[resourceId('Microsoft.Network/publicIPAddresses',concat(variables('publicIPAddressName'), copyIndex()))]"
|
||||
}
|
||||
}
|
||||
}]
|
||||
}
|
||||
},
|
||||
{
|
||||
"apiVersion": "2014-12-01-preview",
|
||||
"type": "Microsoft.Compute/virtualMachines",
|
||||
"name": "[concat(variables('vmName'), copyIndex())]",
|
||||
"location": "[parameters('region')]",
|
||||
"copy": {
|
||||
"name": "nameNodeNicLoop",
|
||||
"count": "[parameters('vmCount')]"
|
||||
},
|
||||
"dependsOn": [
|
||||
"[concat('Microsoft.Network/networkInterfaces/', variables('nicName'), copyIndex())]",
|
||||
"[concat('Microsoft.Storage/storageAccounts/', variables('storageAccountName'), copyIndex())]"
|
||||
],
|
||||
"properties": {
|
||||
"hardwareProfile": {
|
||||
"vmSize": "[parameters('vmSpec').vmSize]"
|
||||
},
|
||||
"osProfile": {
|
||||
"computername": "[concat(variables('vmName'), copyIndex())]",
|
||||
"adminUsername": "[parameters('vmSpec').adminUserName]",
|
||||
"adminPassword": "[parameters('vmSpec').adminPassword]",
|
||||
"linuxConfiguration" : {
|
||||
"disablePasswordAuthentication": "true",
|
||||
"ssh": {
|
||||
"publicKeys": [
|
||||
{
|
||||
"keyData": "[parameters('vmSpec').publicSSHCert]",
|
||||
"path": "[concat('/home/',parameters('vmSpec').adminUserName,'/.ssh/authorized_keys')]"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"storageProfile": {
|
||||
"sourceImage": {
|
||||
"id": "[parameters('vmSpec').sourceImageName]"
|
||||
},
|
||||
"destinationVhdsContainer" : "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/')]",
|
||||
"dataDisks": [
|
||||
{
|
||||
"name": "datadisk0",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 0,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk0.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk1",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 1,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk1.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk2",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 2,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk2.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk3",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 3,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk3.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk4",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 4,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk4.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk5",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 5,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk5.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk6",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 6,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk6.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk7",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 7,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk7.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk8",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 8,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk8.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk9",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 9,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk9.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk10",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 10,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk10.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk11",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 11,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk11.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk12",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 12,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk12.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk13",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 13,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk13.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk14",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 14,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk14.vhd')]"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "datadisk15",
|
||||
"diskSizeGB": "1023",
|
||||
"lun": 15,
|
||||
"vhd": {
|
||||
"Uri": "[concat('http://', variables('storageAccountName'), copyIndex(), '.blob.core.windows.net/vhds/', variables('vmName'), '-datadisk15.vhd')]"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"networkProfile": {
|
||||
"networkInterfaces" : [{
|
||||
"id": "[resourceId('Microsoft.Network/networkInterfaces',concat(variables('nicName'), copyIndex()))]"
|
||||
}]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Microsoft.Compute/virtualMachines/extensions",
|
||||
"name": "[concat(variables('vmName'), copyIndex(), '/prepareDisks')]",
|
||||
"apiVersion": "2014-12-01-preview",
|
||||
"location": "[parameters('region')]",
|
||||
"copy": {
|
||||
"name": "nameNodeNicLoop",
|
||||
"count": "[parameters('vmCount')]"
|
||||
},
|
||||
"dependsOn": [
|
||||
"[concat('Microsoft.Compute/virtualMachines/', variables('vmName'), copyIndex())]"
|
||||
],
|
||||
"properties": {
|
||||
"publisher": "Microsoft.OSTCExtensions",
|
||||
"type": "CustomScriptForLinux",
|
||||
"typeHandlerVersion": "1.2",
|
||||
"settings": {
|
||||
"fileUris": [
|
||||
"[concat(parameters('scriptsUri'), '/scripts/prepareDisks.sh')]",
|
||||
"[concat(parameters('scriptsUri'), '/scripts/initialize-node.sh')]"
|
||||
],
|
||||
"commandToExecute": "sh initialize-node.sh"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,121 @@
|
|||
#!/usr/bin/env bash
|
||||
# Usage: bootstrap-cloudera-1.0.sh {clusterName} {managment_node} {cluster_nodes} {isHA} {sshUserName} [{sshPassword}]
|
||||
|
||||
execname=$0
|
||||
|
||||
log() {
|
||||
echo "[${execname}] $@"
|
||||
}
|
||||
|
||||
log "BEGIN: Processing text stream from Azure ARM call"
|
||||
|
||||
ClusterName=$1
|
||||
ManagementNode=$2
|
||||
ClusterNodes=$3
|
||||
HA=$4
|
||||
User=$5
|
||||
Password=$6
|
||||
|
||||
log "set private key"
|
||||
file="/home/$User/.ssh/id_rsa"
|
||||
key="/tmp/id_rsa.pem"
|
||||
openssl rsa -in $file -outform PEM > $key
|
||||
|
||||
log "remove requiretty"
|
||||
sed -i 's^requiretty^!requiretty^g' /etc/sudoers
|
||||
log "done removing requiretty"
|
||||
|
||||
log "cm ip fix"
|
||||
#CM IP fix. Strips back ticks and creates the format getting the IP address.
|
||||
CM_IP=$(echo $ManagementNode | sed 's/:/ /' | sed 's/:/ /')
|
||||
echo "$CM_IP" >> /etc/hosts
|
||||
|
||||
OIFS=$IFS
|
||||
IFS=':'
|
||||
mip=''
|
||||
for x in $CM_IP
|
||||
do
|
||||
mip=$(echo "$x" | sed 's/:/ /' | sed 's/:/ /' | cut -d ' ' -f 1)
|
||||
log "CM IP: $mip"
|
||||
done
|
||||
IFS=OIFS
|
||||
|
||||
log "Cluster Name: $ClusterName and User Name: $User"
|
||||
|
||||
log "worker name fix"
|
||||
#Worker string fix. Strips back ticks and creates the format for /etc/hosts file
|
||||
Worker_IP=$ClusterNodes
|
||||
log $Worker_IP
|
||||
|
||||
#echo $Worker_IP
|
||||
wip_string=''
|
||||
OIFS=$IFS
|
||||
IFS=','
|
||||
for x in $Worker_IP
|
||||
do
|
||||
log "Workier IP: $x"
|
||||
line=$(echo "$x" | sed 's/:/ /' | sed 's/:/ /')
|
||||
log "New Worker IP to be added to /etc/hosts: $line"
|
||||
echo "$line" >> /etc/hosts
|
||||
wip_string+=$(echo "$line" | cut -d ' ' -f 1 | sed 's/$/,/')
|
||||
log "current wip_string is: $wip_string"
|
||||
done
|
||||
IFS=OIFS
|
||||
worker_ip=$(echo "${wip_string%?}")
|
||||
#echo "$worker_ip"
|
||||
log "Worker ip to be supplied to next script: $worker_ip"
|
||||
|
||||
log "END: processing text stream from Azure ARM call"
|
||||
log "BEGIN: Copy hosts file to all nodes"
|
||||
|
||||
OIFS=$IFS
|
||||
IFS=','
|
||||
|
||||
for node in $ClusterNodes
|
||||
do
|
||||
remote=$(echo "$node" | sed 's/:/ /' | sed 's/:/ /' | cut -d ' ' -f 2)
|
||||
log "Copy hosts file to: $remote"
|
||||
scp -o StrictHostKeyChecking=no -i /home/$User/.ssh/id_rsa /etc/hosts $User@$remote:/tmp/hosts
|
||||
ssh -o StrictHostKeyChecking=no -i /home/$User/.ssh/id_rsa -t -t $User@$remote sudo cp /tmp/hosts /etc/hosts
|
||||
ssh -o StrictHostKeyChecking=no -i /home/$User/.ssh/id_rsa -t -t $User@$remote "sudo bash -c 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'"
|
||||
ssh -o StrictHostKeyChecking=no -i /home/$User/.ssh/id_rsa -t -t $User@$remote "echo vm.swappiness=1 | sudo tee -a /etc/systctl.conf; sudo echo 1 | sudo tee /proc/sys/vm/swappiness"
|
||||
ssh -o StrictHostKeyChecking=no -i /home/$User/.ssh/id_rsa -t -t $User@$remote "sudo ifconfig -a >> initialIfconfig.out; who -b >> initialRestart.out"
|
||||
ssh -o StrictHostKeyChecking=no -i /home/$User/.ssh/id_rsa -t -t $User@$remote "sudo yum install -y ntp; sudo service ntpd start; sudo service ntpd status"
|
||||
done
|
||||
|
||||
sudo yum install -y ntp
|
||||
sudo service ntpd start
|
||||
sudo service ntpd status
|
||||
|
||||
#log "About to format all disks in cluster"
|
||||
#chmod 777 ./diskFormatAndMount.sh
|
||||
#log "Done chmodding run file"
|
||||
|
||||
ClusterNodes=("${ClusterNodes[@]}" $ManagementNode)
|
||||
|
||||
#./diskFormatAndMount.sh ${ClusterNodes[@]}
|
||||
|
||||
log "Just completed formatting all disks in cluster"
|
||||
|
||||
log "END: Copy hosts file to all nodes"
|
||||
|
||||
log "BEGIN: Create Impala Scratch Directories"
|
||||
numDataDirs=$(ls -la / | grep data | wc -l)
|
||||
let endLoopIter=(numDataDirs - 1)
|
||||
|
||||
for node in $Worker_IP
|
||||
do
|
||||
remote=$(echo "$node" | sed 's/:/ /' | sed 's/:/ /' | cut -d ' ' -f 2)
|
||||
log "Creating Impala directories on $remote"
|
||||
ssh -o StrictHostKeyChecking=no -i /home/$User/.ssh/id_rsa -t -t $User@$remote 'numDataDirs=$(ls -la / | grep data | wc -l); let endLoopIter=(numDataDirs - 1); for x in $(seq 0 $endLoopIter); do sudo mkdir -p /data${x}/impala/scratch; sudo chmod 777 /data${x}/impala/scratch; ls -la /data${x}/impala/; done';
|
||||
|
||||
done
|
||||
|
||||
log "END: Create Impala Scratch Directories"
|
||||
|
||||
IFS=$OIFS
|
||||
|
||||
log "BEGIN: Starting detached script to finalize initialization"
|
||||
sh initialize-cloudera-server.sh "$ClusterName" "$key" "$mip" "$worker_ip" $HA $User $Password >/dev/null 2>&1
|
||||
log "END: Detached script to finalize initialization running. PID: $!"
|
||||
|
||||
Разница между файлами не показана из-за своего большого размера
Загрузить разницу
|
|
@ -0,0 +1,100 @@
|
|||
#!/usr/bin/env bash
|
||||
|
||||
execname=$0
|
||||
|
||||
log() {
|
||||
echo "[${execname}] $@" >> /tmp/initialize-cloudera-server.log
|
||||
}
|
||||
|
||||
#fail on any error
|
||||
set -e
|
||||
|
||||
ClusterName=$1
|
||||
key=$2
|
||||
mip=$3
|
||||
worker_ip=$4
|
||||
HA=$5
|
||||
User=$6
|
||||
Password=$7
|
||||
|
||||
log "BEGIN: master node deployments"
|
||||
|
||||
log "Beginning process of disabling SELinux"
|
||||
|
||||
log "Running as $(whoami) on $(hostname)"
|
||||
|
||||
# Use the Cloudera-documentation-suggested workaround
|
||||
log "about to set setenforce to 0"
|
||||
set +e
|
||||
setenforce 0 >> /tmp/setenforce.out
|
||||
|
||||
exitcode=$?
|
||||
log "Done with settiing enforce. Its exit code was $exitcode"
|
||||
|
||||
log "Running setenforce inline as $(setenforce 0)"
|
||||
|
||||
getenforce
|
||||
log "Running getenforce inline as $(getenforce)"
|
||||
getenforce >> /tmp/getenforce.out
|
||||
|
||||
log "should be done logging things"
|
||||
|
||||
|
||||
cat /etc/selinux/config > /tmp/beforeSelinux.out
|
||||
log "ABOUT to replace enforcing with disabled"
|
||||
sed -i 's^SELINUX=enforcing^SELINUX=disabled^g' /etc/selinux/config || true
|
||||
|
||||
cat /etc/selinux/config > /tmp/afterSeLinux.out
|
||||
log "Done disabling selinux"
|
||||
|
||||
set +e
|
||||
|
||||
log "Set cloudera-manager.repo to CM v5"
|
||||
yum clean all >> /tmp/initialize-cloudera-server.log
|
||||
rpm --import http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera >> /tmp/initialize-cloudera-server.log
|
||||
wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo -O /etc/yum.repos.d/cloudera-manager.repo >> /tmp/initialize-cloudera-server.log
|
||||
yum install -y oracle-j2sdk* cloudera-manager-daemons cloudera-manager-server cloudera-manager-server-db* >> /tmp/initialize-cloudera-server.log
|
||||
|
||||
log "start cloudera-scm-server-db and cloudera-scm-server services"
|
||||
service cloudera-scm-server-db start >> /tmp/initialize-cloudera-server.log
|
||||
service cloudera-scm-server start >> /tmp/initialize-cloudera-server.log
|
||||
|
||||
log "Create HIVE metastore DB Cloudera embedded PostgreSQL"
|
||||
export PGPASSWORD=$(head -1 /var/lib/cloudera-scm-server-db/data/generated_password.txt)
|
||||
SQLCMD=( """CREATE ROLE hive LOGIN PASSWORD 'hive';""" """CREATE DATABASE hive OWNER hive ENCODING 'UTF8';""" """ALTER DATABASE hive SET standard_conforming_strings = off;""" )
|
||||
for SQL in "${SQLCMD[@]}"; do
|
||||
psql -A -t -d scm -U cloudera-scm -h localhost -p 7432 -c "${SQL}" >> /tmp/initialize-cloudera-server.log
|
||||
done
|
||||
while ! (exec 6<>/dev/tcp/$(hostname)/7180) ; do log 'Waiting for cloudera-scm-server to start...'; sleep 15; done
|
||||
log "END: master node deployments"
|
||||
|
||||
# Set up python
|
||||
rpm -ivh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm >> /tmp/initialize-cloudera-server.log
|
||||
yum -y install python-pip >> /tmp/initialize-cloudera-server.log
|
||||
pip install cm_api >> /tmp/initialize-cloudera-server.log
|
||||
|
||||
# trap file to indicate done
|
||||
log "creating file to indicate finished"
|
||||
touch /tmp/readyFile
|
||||
|
||||
# Execute script to deploy Cloudera cluster
|
||||
log "BEGIN: CM deployment - starting"
|
||||
logCmd="Command: python cmxDeployOnIbiza.py -n "\""$ClusterName"\"" -u "\""$User"\"" -k "\""$key"\"" -m "\""$mip"\"" -w "\""$worker_ip"\"""
|
||||
if $HA; then
|
||||
logCmd="$logCmd -a"
|
||||
fi
|
||||
log $logCmd
|
||||
if $HA; then
|
||||
python cmxDeployOnIbiza.py -n "$ClusterName" -u $User -k "$key" -m "$mip" -w "$worker_ip" -a >> /tmp/initialize-cloudera-server.log 2>> /tmp/initialize-cloudera-server.err
|
||||
else
|
||||
python cmxDeployOnIbiza.py -n "$ClusterName" -u $User -k "$key" -m "$mip" -w "$worker_ip" >> /tmp/initialize-cloudera-server.log 2>> /tmp/initialize-cloudera-server.err
|
||||
fi
|
||||
|
||||
|
||||
# Sleep for a while to give the agents enough time to check in with the master.
|
||||
# sleep_time=1800
|
||||
# echo "Sleeping for $sleep_time seconds so that script does not report back to Ibiza portal that work is finished."
|
||||
# sleep $sleep_time
|
||||
# echo "Done sleeping. Reporting back to Ibiza portal to let end user know that cluster is coming up."
|
||||
|
||||
log "END: CM deployment ended"
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
#!/bin/bash
|
||||
sh ./initialize-node.sh
|
||||
|
||||
# Put the command line parameters into named variables
|
||||
IPPREFIX=$1
|
||||
NAMEPREFIX=$2
|
||||
NAMESUFFIX=$3
|
||||
NAMENODES=$4
|
||||
DATANODES=$5
|
||||
ADMINUSER=$6
|
||||
|
||||
# Converts a domain like machine.domain.com to domain.com by removing the machine name
|
||||
NAMESUFFIX=`echo $NAMESUFFIX | sed 's/^[^.]*\.//'`
|
||||
|
||||
#use the key from the key vault as the SSH private key
|
||||
openssl rsa -in /var/lib/waagent/*.prv -out /home/$ADMINUSER/.ssh/id_rsa
|
||||
chmod 600 /home/$ADMINUSER/.ssh/id_rsa
|
||||
chown $ADMINUSER /home/$ADMINUSER/.ssh/id_rsa
|
||||
|
||||
#Generate IP Addresses for the cloudera setup
|
||||
NODES=()
|
||||
|
||||
let "NAMEEND=NAMENODES-1"
|
||||
for i in $(seq 0 $NAMEEND)
|
||||
do
|
||||
let "IP=i+10"
|
||||
NODES+=("10.0.0.$IP:${NAMEPREFIX}-nn$i:${NAMEPREFIX}-nn$i.$NAMESUFFIX")
|
||||
done
|
||||
|
||||
let "DATAEND=DATANODES-1"
|
||||
for i in $(seq 0 $DATAEND)
|
||||
do
|
||||
let "IP=i+20"
|
||||
NODES+=("10.0.0.$IP:${NAMEPREFIX}-dn$i:${NAMEPREFIX}-dn$i.$NAMESUFFIX")
|
||||
done
|
||||
|
||||
IFS=',';NODE_IPS="${NODES[*]}";IFS=$' \t\n'
|
||||
|
||||
sh bootstrap-cloudera.sh 'cloudera' "10.0.0.9:${NAMEPREFIX}-mn:${NAMEPREFIX}-mn.$NAMESUFFIX" $NODE_IPS false testuser >> /home/$ADMINUSER/bootstrap-cloudera.log
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Disable the need for a tty when running sudo
|
||||
sed -i '/Defaults[[:space:]]\+!*requiretty/s/^/#/' /etc/sudoers
|
||||
|
||||
# Mount and format the attached disks
|
||||
sh ./prepareDisks.sh
|
||||
|
|
@ -0,0 +1,107 @@
|
|||
#!/bin/bash
|
||||
|
||||
# ok this is the fun part. Let's create a file here
|
||||
cat > inputs2.sh << 'END'
|
||||
|
||||
helloFromInputs() {
|
||||
|
||||
echo "hello from printinputs.sh"
|
||||
}
|
||||
|
||||
printFstab() {
|
||||
echo "Here is the fstab from `hostname`"
|
||||
cat /etc/fstab
|
||||
echo "Now sudo print fstab from `hostname`"
|
||||
sudo cat /etc/fstab
|
||||
}
|
||||
|
||||
|
||||
mountDrive() {
|
||||
|
||||
driveName=$1
|
||||
driveId=$2
|
||||
echo "$(hostname) : /data${2} :About to mount drive"
|
||||
mount -o noatime -t ext4 ${1} /data${2}
|
||||
echo "$driveName /data${2} ext4 defaults,noatime 1 2" | sudo tee -a /etc/fstab
|
||||
cat /etc/fstab
|
||||
echo "$(hostname) : /data${2} : Done mounting drive"
|
||||
|
||||
}
|
||||
|
||||
unmountDrive() {
|
||||
|
||||
driveName=$1
|
||||
driveId=$2
|
||||
echo "$(hostname) : /data${2} : About to unmount drive"
|
||||
umount ${1}
|
||||
df -h
|
||||
echo "$(hostname) : /data${2} : now let's try it with sudo"
|
||||
sudo umount ${1}
|
||||
|
||||
echo "$(hostname) : /data${2} :done trying it with sudo"
|
||||
echo "$(hostname) : /data${2} :Done unmounting drive $(hostname): $1"
|
||||
|
||||
}
|
||||
|
||||
formatAndMountDrive() {
|
||||
echo "$(hostname) : $1 : Beginning operation on drive" || true
|
||||
echo "$(hostname) : $1 : Formatting drive for ext4" || true
|
||||
drive=$1
|
||||
echo "$(hostname) : $1 : set drive and execute"
|
||||
mke2fs -F -t ext4 -b 4096 -O sparse_super,dir_index,extent,has_journal -m1 $drive
|
||||
echo "$(hostname) : $1 : should be done formatting now"
|
||||
|
||||
echo "$(hostname) : $1 : attempt to format exit code: $?"
|
||||
echo "$(hostname) : $1 : Mounting drive $1"
|
||||
echo "$(hostname) : $1 : About to make dir /data${2}"
|
||||
rm -rf /data${2} || true
|
||||
mkdir -p /data${2}
|
||||
chmod 777 /data${2}
|
||||
echo "$(hostname) : $1 : after data creation for id $2: $?"
|
||||
mount -o noatime -t ext4 ${1} /data${2}
|
||||
echo "$driveName /data${2} ext4 defaults,noatime 1 2" | sudo tee -a /etc/fstab
|
||||
echo "$(hostname) : $1 : after mounting for id $2 exit code: $?"
|
||||
echo "$(hostname) : $1 : Done operating on drive $1. Here is df -h"
|
||||
df -h
|
||||
echo "$(hostname) : $1 : done"
|
||||
}
|
||||
|
||||
mountAllDrives() {
|
||||
echo "Mounting all drives"
|
||||
let i=0 || true
|
||||
for x in $(sfdisk -l 2>/dev/null | cut -d' ' -f 2 | grep /dev | grep -v "/dev/sda" | grep -v "/dev/sdb" | sed "s^:^^");
|
||||
do
|
||||
echo "$(hostname) : $x : About to mount drive for $(hostname): $x"
|
||||
mountDrive $x $i
|
||||
let i=(i+1) || true
|
||||
done
|
||||
}
|
||||
|
||||
unmountAllDrives() {
|
||||
let i=0 || true
|
||||
for x in $(sfdisk -l 2>/dev/null | cut -d' ' -f 2 | grep /dev | grep -v "/dev/sda" | grep -v "/dev/sdb" | sed "s^:^^");
|
||||
do
|
||||
echo "$(hostname) : $x : About to call unmountDrive"
|
||||
unmountDrive $x $i 0</dev/null &
|
||||
let i=(i + 1) || true
|
||||
done
|
||||
wait
|
||||
echo "$(hostname) : Done unmounting on $(hostname). Drives look like: "
|
||||
df -h
|
||||
}
|
||||
|
||||
formatAndMountAllDrives() {
|
||||
echo "Entered formatAndMountAllDrives on `hostname`"
|
||||
let i=0 || true
|
||||
for x in $(sfdisk -l 2>/dev/null | cut -d' ' -f 2 | grep /dev | grep -v "/dev/sda" | grep -v "/dev/sdb" | sed "s^:^^");
|
||||
do
|
||||
echo "$(hostname) : $(x): About to call formatAndMountDrive)"
|
||||
formatAndMountDrive $x $i 0</dev/null &
|
||||
let i=(i + 1) || true
|
||||
done
|
||||
wait
|
||||
}
|
||||
END
|
||||
|
||||
bash -c "source ./inputs2.sh; helloFromInputs; printFstab; unmountAllDrives; formatAndMountAllDrives;"
|
||||
exit 0 # and this is useful
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
-----BEGIN RSA PRIVATE KEY-----
|
||||
MIIEowIBAAKCAQEAuM9aF13nZ+HIhP76XtX8Oa5p5whhZ0df6Rmho/fIwe1MB7iu
|
||||
MD7YjbZmIMunB6vx7oou0I7oAh9O79THODOo4jBrFEtfclmG/kEjjXliE7zAEeMi
|
||||
84caSEv4fcWbj5Sf591i6wcakQzqRVi6bMhuRlUFMQPT9XkcGkR9mOrLgHIKl9yR
|
||||
sahp1yBkhWBCcdVJB2BI4Tmb5keIbAhQsk9DSqWG2VTlH2vCC80oQDcHgJvplsWK
|
||||
88LCBO1qsuS9+WJjGcl4ADgQtZeKa8Uo9obdC7sncBZHOCvTXHvmdtCOiV4ZIDwU
|
||||
wEpAaPsA7ppqt4/FpuAizVAdiD3Ln8HfwdFahwIDAQABAoIBABXnBR9V0rH+ETyp
|
||||
11Rb+euAHy7xUFHkJJrEneKH4DQTgno7yfEP0axh3JKwcBg5qY+Csu+msBpoSXuk
|
||||
9QNVJAPChz49fI9ZyuThJWyD4TrDmOPFzoOcFGJTFp0ZSUln7NeW5j1Cd0SPikgz
|
||||
rsxrqJ4PP3FCqeEivFDkoxn3YazUI3SyMJMFZMcQxoh2q+5oSStXliw4LnYawV4T
|
||||
2jIhlNNU/W0vtp2OITJh1QvCGOFDHo0TJ63Jqhfg8P28jZ/bZaXd4beiDBB/F+Iq
|
||||
Y1X9IpCiRfNxRj/ZG5MYAgZ9cU5shxiYaaNhouamSsSQNvw2EQqqF7X0AYg8jnOq
|
||||
GTEGC1kCgYEA6x9AzZbKbJN7QIo7qGiKf0o3nDRmuEp0H2UOVK8azWIAu6c532eH
|
||||
OjCCo0mDcDifnO6eSbj0Mrc0nl0p9eUZbAB7mQt4x6wy5ivBVHXnvAF19NFZk3iU
|
||||
nA3gRJdHU2b2gqGRkdIJfKnCp4SjeMESGQ6q/yzCse/fAyOAP1/NMhsCgYEAyThq
|
||||
BUKE30nbNUbxstHQOVIUfG2YS/vD3t1XQIls20N0Wb3iVnYc7vBArub5k609EmkD
|
||||
AcifTJMiZ/uAEWbQ6I4H6R+rOQAPbXZKgz5jt4r7+XGj9ZkuOYI8q7Y4yE5pYi7r
|
||||
TZNGQnf0TmCJ5UJc78acW34mzMk/N8A8D55kIAUCgYAVjQTMk1C0iQQYE1OKiSKA
|
||||
rlXQjIEJzVE9e27H1lD3mE8G5DoKvxuG+uFOhDN2+smugbSif0Bq0Z4SEeoBruEY
|
||||
pCawDXwhmSgFLueGjVQTSHL+YJ38RWYhs6flwZP5EgvwR0znd8AE3C6KWj79l3zI
|
||||
LS8BOqIR2zGuwPdFV8ugFwKBgGDFTseky/4OqOugaNDU8sz+WnfUxS+VrwdZb4LM
|
||||
FJwGfshB974ZI6UNyi3TSyfoYryd4CJVd7b988gKPfcgeUu+MrO81T2ed8zu3Lrt
|
||||
oEs+/htiubV9q6tgpQj3o6tBepELe4JdV6bTxZyD9Q1asxNlcwDVVJecGVkYjEn+
|
||||
pKx5AoGBAIHym5qlxFZFebIU94MnnJolZiwZ0MRP5qtkRll0KczAN8XuMxbANBmM
|
||||
ajIEVuZ/cEnsZ15bhMY7pG7n0Xgzx+IBHRc7CfEedjwndtixsduJv72hIWWnQp2v
|
||||
KVZnzQLnDmbuFPxBxTe/8LcmYEP29FzEFw4EysEie8A56npfTGWs
|
||||
-----END RSA PRIVATE KEY-----
|
||||
Двоичный файл не отображается.
|
|
@ -0,0 +1,38 @@
|
|||
{
|
||||
"$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
|
||||
"contentVersion": "1.0.0.0",
|
||||
"parameters": {
|
||||
"region": {
|
||||
"type": "string"
|
||||
},
|
||||
"networkSpec": {
|
||||
"type": "object"
|
||||
}
|
||||
},
|
||||
"variables": {
|
||||
},
|
||||
"resources": [
|
||||
{
|
||||
"apiVersion": "2014-12-01-preview",
|
||||
"type": "Microsoft.Network/virtualNetworks",
|
||||
"name": "[parameters('networkSpec').virtualNetworkName]",
|
||||
"location": "[parameters('region')]",
|
||||
"properties": {
|
||||
"addressSpace": {
|
||||
"addressPrefixes": [
|
||||
"[parameters('networkSpec').addressPrefix]"
|
||||
]
|
||||
},
|
||||
"subnets": [
|
||||
{
|
||||
"name": "[parameters('networkSpec').virtualNetworkSubnetName]",
|
||||
"properties": {
|
||||
"addressPrefix": "[parameters('networkSpec').subnetPrefix]"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
Param(
|
||||
[string]$resourceGroupName,
|
||||
[string]$region,
|
||||
[string]$keyVaultName,
|
||||
[string]$keyName,
|
||||
[string]$pfxFile
|
||||
)
|
||||
|
||||
# Import the key vault management scripts, will most likely become part of the Azure PowerShell tools at some point
|
||||
# import-module .\KeyVaultManager
|
||||
|
||||
# Switch to ARM mode in PowerShell
|
||||
Switch-AzureMode -Name AzureResourceManager
|
||||
|
||||
# Create a new Util resource group in the correct region
|
||||
$resourceGroup = Get-AzureResourceGroup -ResourceGroupName $resourceGroupName 2>$null
|
||||
if (!$resourceGroup) {
|
||||
Write-Host "Creating new resource group with name '$resourceGroupName'."
|
||||
New-AzureResourceGroup -Name $resourceGroupName -Location $region
|
||||
}
|
||||
else {
|
||||
Write-Host "Resource group '$resourceGroupName' already exists."
|
||||
}
|
||||
|
||||
# Create a new key vault in the Util resource group
|
||||
$keyVault = Get-AzureKeyVault -VaultName $keyVaultName 2>$null
|
||||
if (!$keyVault) {
|
||||
Write-Host "Creating new key vault with name '$keyVaultName'."
|
||||
New-AzureKeyVault -VaultName $keyVaultName -ResourceGroupName $resourceGroupName -Location $region -EnabledForDeployment
|
||||
}
|
||||
else {
|
||||
Write-Host "Key vault '$keyVaultName' already exists."
|
||||
}
|
||||
|
||||
# Format and upload the .pfx certificate to the Key Vault
|
||||
$fileContentBytes = get-content $pfxFile -Encoding Byte
|
||||
$fileContentEncoded = [System.Convert]::ToBase64String($fileContentBytes)
|
||||
|
||||
$jsonObject = @"
|
||||
{
|
||||
"data": "$filecontentencoded",
|
||||
"dataType" :"pfx",
|
||||
"password": ""
|
||||
}
|
||||
"@
|
||||
|
||||
$jsonObjectBytes = [System.Text.Encoding]::UTF8.GetBytes($jsonObject)
|
||||
$jsonEncoded = [System.Convert]::ToBase64String($jsonObjectBytes)
|
||||
|
||||
$secret = ConvertTo-SecureString -String $jsonEncoded -AsPlainText -Force
|
||||
$key = Set-AzureKeyVaultSecret -VaultName $keyVaultName -Name $keyName -SecretValue $secret
|
||||
$url = $key.Id
|
||||
|
||||
Write-Host "Key has been uploaded successfully"
|
||||
Write-Host " - Url: $url"
|
||||
Загрузка…
Ссылка в новой задаче