Add CLI Singularity image

- Number remote fs disks starting at 0

CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ## [Unreleased]
 
+### Added
+- CLI Singularity image (#135)
+
+### Changed
+- Start LUN numbering for remote fs disks at 0
+
 ## [3.0.0rc1] - 2017-11-08
 ### Changed
 - Update VM size support

convoy/remotefs.py

@@ -484,7 +484,7 @@ def _create_virtual_machine(
     vm_name = settings.generate_virtual_machine_name(
         rfs.storage_cluster, offset)
     # construct data disks array
-    lun = 3
+    lun = 0
     data_disks = []
     for diskname in rfs.storage_cluster.vm_disk_map[offset].disk_array:
         data_disks.append(

docker/linux/cli/Dockerfile

@@ -14,9 +14,9 @@ RUN apk update \
     && cd /opt/batch-shipyard \
     && git checkout $GIT_COMMIT \
     && rm -rf .git \
-    && rm -f .git* .travis.yml appveyor.yml install* \
+    && rm -f .git* .travis.yml *.yml install* \
     && pip3 install --no-cache-dir -r requirements.txt \
-    && python3 -m compileall -f /opt/batch-shipyard \
+    && python3 -m compileall -f /opt/batch-shipyard/shipyard.py /opt/batch-shipyard/convoy \
     && apk del --purge build-base python3-dev libressl-dev libffi-dev git \
     && rm /var/cache/apk/*
 

docker/windows/cli/Dockerfile

@@ -18,11 +18,12 @@ RUN git clone -b $Env:GIT_BRANCH --single-branch --depth 5 https://github.com/Az
     Remove-Item .git -Force -Recurse ; \
     Remove-Item .git* -Force -Recurse ; \
     Remove-Item .travis.yml -Force -Recurse ; \
-    Remove-Item appveyor.yml -Force -Recurse ; \
+    Remove-Item *.yml -Force -Recurse ; \
     Remove-Item install* -Force -Recurse
 
 RUN python -m compileall C:\Python\Lib ; \
-    python -m compileall C:\batch-shipyard ; \
+    python -m compileall C:\batch-shipyard\shipyard.py ; \
+    python -m compileall C:\batch-shipyard\convoy ; \
 	exit 0
 
 FROM microsoft/nanoserver

docs/00-introduction.md

@@ -18,7 +18,9 @@ assoicated datacenter data egress and storage costs, i.e., the same cost
 as if consuming
 [Virtual Machines](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/)
 or [Cloud Services](https://azure.microsoft.com/en-us/pricing/details/cloud-services/)
-directly.
+directly. Additionally, Azure Batch provides the ability to run on
+[Low Priority VMs](https://docs.microsoft.com/en-us/azure/batch/batch-low-pri-vms)
+which can dramatically lower costs for savings up to 80%!
 
 Azure Batch can handle workloads on any point of the parallel and distributed
 processing spectrum, from embarassingly parallel workloads all the way to
@@ -110,8 +112,8 @@ their respective packaging and tooling, users can spend less time hassling
 with the underlying infrastructure, VM application state consistency,
 potential dependency interaction side effects and spend more
 time on things that actually matter for their batch workloads: the job and
-task results themselves. And with Azure Batch, you can scale your workload up
-and only pay for the compute resources you use. The Batch Shipyard
+task results themselves. And with Azure Batch, you can automatically scale
+your workload and only pay for the compute resources you use. Batch Shipyard
 provides a way to combine your containers with the power of scale-out cloud
 computing with ease!
 

docs/01-batch-shipyard-installation.md

@@ -6,6 +6,7 @@ pick an option that is most suitable for your work environment.
 * [Pre-built binary](#binary)
 * [Installers](#installers)
 * [Docker image](#docker-install)
+* [Singularity image](#singularity-install)
 * [Jupyter Notebooks](#jupyter)
 
 If you wish to install Batch Shipyard into your Azure App Service (e.g.,
@@ -60,8 +61,9 @@ to your operating system specific installation instructions:
 * [Windows Subsystem for Linux](#wsl-install)
 
 Alternatively, you can install the Batch Shipyard CLI on your machine via
-[Docker](#docker-install). If using the Docker image, this is the only step
-needed and does not require any futher installation steps.
+[Docker](#docker-install) or [Singularity](#singularity-install). If using
+the Docker image, this is the only step needed and does not require any
+futher installation steps.
 
 ### <a name="linux-install"></a>Step 2 [Linux]: Run the `install.sh` Script
 Batch Shipyard includes an installation script to simplify installation on
@@ -317,11 +319,28 @@ To install:
 ```shell
 docker pull alfpark/batch-shipyard:latest-cli
 ```
-This will pull the cli Docker image of batch-shipyard to your local machine.
+This will pull the CLI Docker image of Batch Shipyard to your local machine.
 You are now ready to execute it with `docker run`. Please see the
 [Batch Shipyard Usage](20-batch-shipyard-usage.md) guide for more information
 on how to execute the Batch Shipyard CLI Docker image.
 
+## <a name="singularity-install"></a>Batch Shipyard CLI Installation via Singularity
+If using the [alfpark/batch-shipyard-singularity:cli](https://www.singularity-hub.org/collections/204)
+Singularity image, then all of the required software is bundled in the image
+itself, however, you will need an installation of Singularity on your
+machine. More information on installing Singularity to your local machine
+can be found [here](http://singularity.lbl.gov/).
+
+To install:
+```shell
+singularity pull shub://alfpark/batch-shipyard-singularity:cli
+```
+This will pull the CLI Singularity image of Batch Shipyard to your local
+machine (to the current working directory). You are now ready to execute
+it with `singularity run` or simply just executing the image. Please see
+the [Batch Shipyard Usage](20-batch-shipyard-usage.md) guide for more
+information on how to execute the Batch Shipyard CLI Singularity image.
+
 ## <a name="jupyter"></a>Jupyter Notebooks
 There are community contributed [Jupyter notebooks](../contrib/notebooks) to
 help you quickly get started if you prefer that environment instead of a

docs/20-batch-shipyard-usage.md

@@ -1,8 +1,8 @@
 # Batch Shipyard Usage
 This page contains in-depth details on how to use the Batch Shipyard tool.
-Please see the [Batch Shipyard Docker Image CLI](#docker-cli) section for
-information regarding how to use the `alfpark/batch-shipyard:latest-cli`
-Docker image if not invoking the Python script directly.
+Please see the [Container Image CLI](#container-cli) section for information
+regarding how to use the Docker or Singularity image if not invoking the
+Python script or pre-built binary directly.
 
 ## Batch Shipyard Invocation
 If you installed Batch Shipyard using the `install.sh` script, then
@@ -646,13 +646,20 @@ The above invocation will stream the stdout.txt file from the job `job1` and
 task `task1` from a live compute node. Because all portions of the
 `--filespec` option are specified, the tool will not prompt for any input.
 
-## <a name="docker-cli"></a>Batch Shipyard Docker Image CLI Invocation
-If using the [alfpark/batch-shipyard:latest-cli](https://hub.docker.com/r/alfpark/batch-shipyard)
-Docker image, then you would invoke the tool as:
+## <a name="container-cli"></a>Batch Shipyard Container Image CLI Invocation
+If using either the Docker image [alfpark/batch-shipyard:latest-cli](https://hub.docker.com/r/alfpark/batch-shipyard)
+or the Singularity image
+[shub://alfpark/batch-shipyard-singularity:cli](https://www.singularity-hub.org/collections/204),
+then you would invoke Batch Shipyard as:
 
 ```shell
+# if using Docker
 docker run --rm -it alfpark/batch-shipyard:latest-cli \
     <command> <subcommand> <options...>
+
+# if using Singularity
+singularity run shub://alfpark/batch-shipyard-singularity:cli \
+    <command> <subcommand> <options...>
 ```
 
 where `<command> <subcommand>` is the command and subcommand as described
@@ -661,29 +668,37 @@ above and `<options...>` are any additional options to pass to the
 
 Invariably, you will need to pass config files to the tool which reside
 on the host and not in the container by default. Please use the `-v` volume
-mount option with `docker run` to mount host directories inside the container.
-For example, if your Batch Shipyard configs are stored in the host path
-`/home/user/batch-shipyard-configs` you could modify the docker run command
-as:
+mount option with `docker run` or `-B` bind option with `singularity run`
+to mount host directories inside the container. For example, if your Batch
+Shipyard configs are stored in the host path
+`/home/user/batch-shipyard-configs` you could modify the invocations as:
 
 ```shell
+# if using Docker
 docker run --rm -it \
     -v /home/user/batch-shipyard-configs:/configs \
-    -e SHIPYARD_CONFIGDIR=/configs \
+    -w /configs \
     alfpark/batch-shipyard:latest-cli \
     <command> <subcommand> <options...>
+
+# if using Singularity
+singularity run \
+    -B /home/user/batch-shipyard-configs:/configs \
+    --pwd /configs \
+    shub://alfpark/batch-shipyard-singularity:cli \
+    <command> <subcommand> <options...>
 ```
 
-Notice that we specified a Docker environment variable via
-`-e SHIPYARD_CONFIGDIR` to match the container path of the volume mount.
+Notice that we specified the working directory as `-w` for Docker or
+`--pwd` for Singularity to match the `/configs` container path.
 
 Additionally, if you wish to ingress data from locally accessible file
 systems using Batch Shipyard, then you will need to map additional volume
 mounts as appropriate from the host to the container.
 
 Batch Shipyard may generate files with some actions, such as adding a SSH
-user or creating a pool with an SSH user. In this case, you will need
-to create a volume mount with the `-v` option and also ensure that the
+user or creating a pool with an SSH user. In this case, you will need to
+create a volume mount with the `-v` (or `-B`) option and also ensure that the
 pool specification `ssh` object has a `generated_file_export_path` property
 set to the volume mount path. This will ensure that generated files will be
 written to the host and persisted after the docker container exits. Otherwise,