update data ingesetor (#114)

docs/getting_data_onto_vfxt.md

@@ -15,11 +15,45 @@ This section explains strategies for creating a multi-client, multithreaded file
 
 It also explains some utilities that can help. The ``msrsync`` utility can be used to partially automate the process of dividing a dataset into buckets and using rsync commands. The ``parallelcp`` script is another utility that reads the source directory and issues copy commands automatically.  
 
-To install a data ingestor VM with all of these parallel data ingestion tools installed, click the "Deploy to Azure" button:
+To install a data ingestor VM with all of these parallel data ingestion tools installed, we will have the client VM pull and run the data ingestor install script from the Avere vFXT mount.  Before deploying the client VM, first setup the Avere vFXT with the install file:
 
-<a href="https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAvere%2Fmaster%2Fsrc%2Fdataingestor%2Fdataingestor-azuredeploy.json" target="_blank">
-<img src="https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/1-CONTRIBUTION-GUIDE/images/deploytoazure.png"/>
-</a>
+1. If you have not already done so, ssh to the controller, and mount to the Avere vFXT:
+
+    1. Run the following commands:
+        ```bash
+        sudo -s
+        apt-get update
+        apt-get install nfs-common
+        mkdir -p /nfs/node0
+        chown nobody:nogroup /nfs/node0
+        ```
+
+    2. Edit `/etc/fstab` to add the following lines but *using your vFXT node IP addresses*. Add more lines if your cluster has more than three nodes.
+        ```bash
+        10.0.0.12:/msazure	/nfs/node0	nfs hard,nointr,proto=tcp,mountproto=tcp,retry=30 0 0
+        ```
+
+    3. To mount all shares, type `mount -a`
+
+2. On the controller, download the dataingestor bootstrap script:
+    ```bash
+    mkdir -p /nfs/node0/bootstrap
+    cd /nfs/node0/bootstrap
+    curl --retry 5 --retry-delay 5 -o /nfs/node0/bootstrap/bootstrap.dataingestor.sh https://raw.githubusercontent.com/Azure/Avere/master/src/dataingestor/bootstrap.dataingestor.sh
+    ```
+
+3. From your controller, verify your dataingestor setup by running the following verify script.  If the script shows success, you are ready to deploy.  Otherwise you will need to fix each error listed.
+
+    ```bash
+    curl -o- https://raw.githubusercontent.com/Azure/Avere/master/src/dataingestor/dataingestorVerify.sh | bash
+    ```
+
+4. Deploy the clients by clicking the "Deploy to Azure" button below, but set the following settings:
+  * specify `/bootstrap/bootstrap.dataingestor.sh` for the bootstrap script
+
+    <a href="https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2FAvere%2Fmaster%2Fsrc%2Fclients%2Fvmas%2Fazuredeploy.json" target="_blank">
+    <img src="https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/1-CONTRIBUTION-GUIDE/images/deploytoazure.png"/>
+    </a>
 
 Click the link to jump to a section:
 

src/clients/README.md

@@ -10,20 +10,20 @@ The clients are mounted roundrobin across the Avere vFXT vServer IP addresses do
 sudo -s
 apt-get update
 apt-get install nfs-common
+mkdir -p /nfs/node0
 mkdir -p /nfs/node1
 mkdir -p /nfs/node2
-mkdir -p /nfs/node3
+chown nobody:nogroup /nfs/node0
 chown nobody:nogroup /nfs/node1
 chown nobody:nogroup /nfs/node2
-chown nobody:nogroup /nfs/node3
 ```
 
 2. Edit `/etc/fstab` to add the following lines but *using your vFXT node IP addresses*. Add more lines if your cluster has more than three nodes.
 
 ```bash
-10.0.0.12:/msazure	/nfs/node1	nfs hard,nointr,proto=tcp,mountproto=tcp,retry=30 0 0
-10.0.0.13:/msazure	/nfs/node2	nfs hard,nointr,proto=tcp,mountproto=tcp,retry=30 0 0
-10.0.0.14:/msazure	/nfs/node3	nfs hard,nointr,proto=tcp,mountproto=tcp,retry=30 0 0
+10.0.0.12:/msazure	/nfs/node0	nfs hard,nointr,proto=tcp,mountproto=tcp,retry=30 0 0
+10.0.0.13:/msazure	/nfs/node1	nfs hard,nointr,proto=tcp,mountproto=tcp,retry=30 0 0
+10.0.0.14:/msazure	/nfs/node2	nfs hard,nointr,proto=tcp,mountproto=tcp,retry=30 0 0
 ```
 
 3. To mount all shares, type `mount -a` from the cluster controller, and then run the following to download the bootstrap script.

src/dataingestor/dataingestorVerify.sh

@@ -1,8 +1,7 @@
 #!/bin/bash
 
 TARGETDIR=/nfs/node0/bootstrap
-VDBENCHINSTALL=$TARGETDIR/bootstrap.vdbench.sh
-VDBENCHSRC="$TARGETDIR/vdbench*.zip"
+DATAINGESTORINSTALL=$TARGETDIR/bootstrap.dataingestor.sh
 
 if [ ! -d "$TARGETDIR" ]; then
     echo "ERROR: directory $TARGETDIR does not exist"
@@ -11,14 +10,8 @@ else
     echo "SUCCESS: $TARGETDIR found"
 fi
 
-if ! ls $VDBENCHINSTALL > /dev/null 2>&1; then
-    echo "MISSING: $VDBENCHINSTALL.  Please download the install script per instructions."
+if ! ls $DATAINGESTORINSTALL > /dev/null 2>&1; then
+    echo "MISSING: $DATAINGESTORINSTALL.  Please download the install script per instructions."
 else
-    echo "SUCCESS: $VDBENCHINSTALL found"
-fi
-
-if ! ls $VDBENCHSRC > /dev/null 2>&1; then
-    echo "MISSING: $VDBENCHSRC.  Please download the vdbench zip file from Oracle."
-else
-    echo "SUCCESS: $VDBENCHSRC found"
-fi
+    echo "SUCCESS: $DATAINGESTORINSTALL found"
+fi