Bug fix: Yarn resource manager connection. (#324)

* Spark on DLWorkspace

* Bug fix, yarn resource manager has connection issue with hadoop name node

docs/deployment/spark.md

@@ -2,9 +2,15 @@
 
 The document describes the procedure to run a spark job on a DL Workspace cluster. Please note that the procedure below will be significantly updated in a future release to stream line the process. 
 
-1. Build Spark docker
+1. Launch Yarn resource manager and node manager
   ```
-  deploy.py docker push spark
+  deploy.py kubernetes start yarnresourcemanager
+  deploy.py kubernetes start yarnnodemanager
+  ```
+  You may disable yarn resource manager and node manager by running:
+  ```
+  deploy.py kubernetes stop yarnresourcemanager
+  deploy.py kubernetes stop yarnnodemanager
   ```
 
 2. Launch Spark container
@@ -12,13 +18,18 @@ The document describes the procedure to run a spark job on a DL Workspace cluste
   deploy.py kubernetes start spark
   ```
 
-3. Ssh into spark container, go to spark directory
+3. Launch Spark container
+  ```
+  deploy.py kubernetes start spark
+  ```
+
+4. Ssh into spark container, go to spark directory
   ```
   deploy.py kubectl exec -ti spark-pod
   cd /usr/local/spark/bin
   ```
 
-4. You should be able to execute spark command, e.g., 
+5. You should be able to execute spark command, e.g., 
   ```
   run-example SparkPi
   ```

src/ClusterBootstrap/services/yarnresourcemanager/yarnrm1.yaml

@@ -32,6 +32,8 @@ spec:
           mountPath: {{cnf["hdfsconfig"]["namenode"]["data"]}} 
         - name: localnamenode
           mountPath: {{cnf["hdfsconfig"]["namenode"]["localdata"]}} 
+        - name: etchosts
+          mountPath: /etc/hosts
         command:
         - sh
         - -c
@@ -51,4 +53,7 @@ spec:
           path: {{cnf["storage-mount-path"]}}/namenodeshare
       - name: localnamenode
         hostPath:
-          path: {{cnf["hdfsconfig"]["namenode"]["localdata"]}} 
+          path: {{cnf["hdfsconfig"]["namenode"]["localdata"]}} 
+      - name: etchosts
+        hostPath:
+          path: /dev/null

src/ClusterBootstrap/services/yarnresourcemanager/yarnrm2.yaml

@@ -32,6 +32,8 @@ spec:
           mountPath: {{cnf["hdfsconfig"]["namenode"]["data"]}}
         - name: localnamenode
           mountPath: {{cnf["hdfsconfig"]["namenode"]["localdata"]}}
+        - name: etchosts
+          mountPath: /etc/hosts
         command:
         - sh
         - -c
@@ -52,3 +54,6 @@ spec:
       - name: localnamenode
         hostPath:
           path: {{cnf["hdfsconfig"]["namenode"]["localdata"]}}
+      - name: etchosts
+        hostPath:
+          path: /dev/null

src/docker-images/spark/bootstrap_hdfs.py

@@ -73,9 +73,16 @@ datanode:    Launch datanode.
 #           f.close()
 #           print logging_config
 #           logutils.dictconfig.dictConfig(logging_config)
-        utils.render_template("hdfs-site.xml.in-docker", "/usr/local/hadoop/etc/hadoop/hdfs-site.xml",config, verbose=verbose)
+        isHA = "namenode2" in config["namenode"]
+        if isHA:
+            utils.render_template("hdfs-site.xml.in-docker", "/usr/local/hadoop/etc/hadoop/hdfs-site.xml",config, verbose=verbose)
+        else:
+            utils.render_template("hdfs-site-single.xml.in-docker", "/usr/local/hadoop/etc/hadoop/hdfs-site.xml",config, verbose=verbose)
         utils.render_template("mapred-site.xml.in-docker", "/usr/local/hadoop/etc/hadoop/mapred-site.xml",config, verbose=verbose)
-        utils.render_template("yarn-site.xml.in-docker", "/usr/local/hadoop/etc/hadoop/yarn-site.xml",config, verbose=verbose)
+        if isHA:
+            utils.render_template("yarn-site.xml.in-docker", "/usr/local/hadoop/etc/hadoop/yarn-site.xml",config, verbose=verbose)
+        else:
+            utils.render_template("yarn-site-single.xml.in-docker", "/usr/local/hadoop/etc/hadoop/yarn-site.xml",config, verbose=verbose)
         utils.render_template("spark-core-site.xml.in-docker", "/usr/local/spark/yarn-remote-client/core-site.xml",config, verbose=verbose)
         utils.render_template("spark-yarn-site.xml.in-docker", "/usr/local/spark/yarn-remote-client/yarn-site.xml",config, verbose=verbose)
     except Exception as e: