Tag for 3.3.0 release

CHANGELOG.md

@@ -2,21 +2,44 @@
 
 ## [Unreleased]
 
+## [3.3.0] - 2018-03-01
 ### Added
-- `storage sas create` command added as a utility helper function to create
-SAS tokens for given storage accounts in credentials
+- Support for specifying default task exit conditions (i.e., non-zero exit
+codes). Please see the jobs configuration doc for more information.
+- New commands (please see usage doc for more information):
+    - `pool nodes prune` will remove all unused Docker-related data on all
+      nodes in the pool (requires an SSH user)
+    - `pool nodes ps` performs a `docker ps -a` on all nodes in the pool
+      (requires an SSH user)
+    - `pool nodes zap` will kill (and optionally remove) **all** running
+      Docker containers on nodes in a pool (requires an SSH user). Note that
+      `jobs tasks term` is the preferred command to control individual
+      (or grouped) task termination or `jobs term --termtasks` to terminate
+      at the job level.
+    - `storage sas create` command added as a utility helper function to
+      create SAS tokens for given storage accounts in credentials
 - Support for activating
 [Azure Hybrid Use Benefit](https://azure.microsoft.com/pricing/hybrid-benefit/)
 for Windows pools
 
 ### Changed
-- Expand node error details
+- Greatly expand pool, node, job, and task details for `list` sub-commands
+- Expand error detail key/value pairs if present
 - Name resources for RemoteFS with 3 digits (e.g., 000 instead of 0) for
 improved alpha ordering
+- Move site extension to nuget.org (#172)
 
 ### Fixed
-- Remote FS cluster del with disks error
+- Command lines in non-native mode now properly expand environment variables
+with normal quoting
+- RemoteFS cluster del with disks error
+- RemoteFS cluster status decoding issues
 - Unintended interaction between native mode and custom images
+- Handle starting Docker service for all deployments
+- Do not automatically mount storage cluster or custom linux mounts on boot
+due to potential race conditions with the ephemeral disk mount
+- Fix TLS issue with powershell (#171)
+- Fix conflicts when using install.sh with Anaconda environments
 - Update packer scripts and fix typos
 - Minor doc updates
 
@@ -24,18 +47,20 @@ improved alpha ordering
 ### Added
 - Custom Linux Mount support for `shared_data_volumes`. Please see the
 global configuration doc for more information.
-- `account` command added with the following sub-commands (requires AAD auth):
-    - `info` provides information about a Batch account (including account
-      level quotas)
-    - `list` provides information about all (or a resource group subset) of
-      accounts within the subscription specified in credentials
-    - `quota` provides service level quota information for the subscription
-      for a given location
+- New commands (please see usage doc for more information):
+    - `account` command added with the following sub-commands (requires
+      AAD auth):
+        - `info` provides information about a Batch account (including account
+          level quotas)
+        - `list` provides information about all (or a resource group subset)
+          of accounts within the subscription specified in credentials
+        - `quota` provides service level quota information for the
+          subscription for a given location
+    - `pool rdp` sub-command added, please see usage doc for more information.
+      Requires Batch Shipyard executing on Windows with target Windows
+      containers pools.
 - `pool images update` command now supports updating Docker images
 in native container support pools via SSH
-- `pool rdp` sub-command added, please see usage doc for more information.
-Requires Batch Shipyard executing on Windows with target Windows containers
-pools.
 - Ability to specify an AAD authority URL via the `aad`:`authority_url`
 credential configuration, `--aad-authority-url` command line option or
 `SHIPYARD_AAD_AUTHORITY_URL` environment variable. Please see relevant
@@ -1139,7 +1164,8 @@ transfer is disabled
 #### Added
 - Initial release
 
-[Unreleased]: https://github.com/Azure/batch-shipyard/compare/3.2.0...HEAD
+[Unreleased]: https://github.com/Azure/batch-shipyard/compare/3.3.0...HEAD
+[3.3.0]: https://github.com/Azure/batch-shipyard/compare/3.2.0...3.3.0
 [3.2.0]: https://github.com/Azure/batch-shipyard/compare/3.1.0...3.2.0
 [3.1.0]: https://github.com/Azure/batch-shipyard/compare/3.0.3...3.1.0
 [3.0.3]: https://github.com/Azure/batch-shipyard/compare/3.0.2...3.0.3

README.md

@@ -17,13 +17,6 @@ Additionally, Batch Shipyard provides the ability to provision and manage
 entire [standalone remote file systems (storage clusters)](http://batch-shipyard.readthedocs.io/en/latest/65-batch-shipyard-remote-fs/)
 in Azure, independent of any integrated Azure Batch functionality.
 
-Batch Shipyard is now integrated directly into
-[Azure Cloud Shell](https://docs.microsoft.com/azure/cloud-shell/overview)
-and you can execute any Batch Shipyard workload using your web browser or
-the Microsoft Azure [Android](https://play.google.com/store/apps/details?id=com.microsoft.azure&hl=en)
-and [iOS](https://itunes.apple.com/us/app/microsoft-azure/id1219013620?mt=8)
-app.
-
 ## Major Features
 * Automated [Docker Host Engine](https://www.docker.com) and
 [Singularity](http://singularity.lbl.gov/) installations tuned for
@@ -79,10 +72,10 @@ on [Azure N-Series VM instances](https://docs.microsoft.com/azure/virtual-machin
 applications packaged in Docker or Singularity on compute pools with
 automatic job completion and task termination
 * Transparent assist for running Docker and Singularity containers utilizing
-Infiniband/RDMA for MPI on HPC low-latency Azure VM instances:
-    * [A-Series](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-hpc): STANDARD\_A8, STANDARD\_A9
-    * [H-Series](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-hpc): STANDARD\_H16R, STANDARD\_H16MR
-    * [N-Series](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-gpu): STANDARD\_NC24R, STANDARD\_NC24RS\_V2, STANDARD\_ND24RS
+Infiniband/RDMA for MPI on HPC low-latency Azure VM instances including
+[A-Series](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-hpc),
+[H-Series](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-hpc),
+and [N-Series](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-gpu)
 * Support for [Azure Batch task dependencies](https://azure.microsoft.com/documentation/articles/batch-task-dependencies/)
 allowing complex processing pipelines and DAGs
 * Support for merge or final task specification that automatically depends
@@ -96,20 +89,27 @@ optional creation of SSH tunneling scripts to Docker Hosts on compute nodes
 [Azure KeyVault](https://azure.microsoft.com/services/key-vault/)
 * Support for [custom host images](http://batch-shipyard.readthedocs.io/en/latest/63-batch-shipyard-custom-images/)
 * Support for [Windows Containers](https://docs.microsoft.com/virtualization/windowscontainers/about/)
-on compliant Windows compute node pools
+on compliant Windows compute node pools with the ability to activate
+[Azure Hybrid Use Benefit](https://azure.microsoft.com/pricing/hybrid-benefit/)
+if applicable
 
 ## Installation
 ### Azure Cloud Shell
-Batch Shipyard is now integrated into Azure Cloud Shell with no installation
-required. Simply request a Cloud Shell session and type `shipyard` to invoke
-the CLI.
+Batch Shipyard is integrated directly into
+[Azure Cloud Shell](https://docs.microsoft.com/azure/cloud-shell/overview)
+and you can execute any Batch Shipyard workload using your web browser or
+the Microsoft Azure [Android](https://play.google.com/store/apps/details?id=com.microsoft.azure&hl=en)
+and [iOS](https://itunes.apple.com/us/app/microsoft-azure/id1219013620?mt=8)
+app.
 
-Try Batch Shipyard now from your browser:
+Simply request a Cloud Shell session and type `shipyard` to invoke the CLI;
+no installation is required. Try Batch Shipyard now from your browser:
 [![Launch Cloud Shell](https://shell.azure.com/images/launchcloudshell.png "Launch Cloud Shell")](https://shell.azure.com)
 
 ### Local Installation
 Please see [the installation guide](http://batch-shipyard.readthedocs.io/en/latest/01-batch-shipyard-installation/)
-for more information regarding installation and requirements.
+for more information regarding the various local installation options and
+requirements.
 
 ## Documentation and Recipes
 Please refer to the
@@ -120,7 +120,7 @@ Visit the
 section for various sample container workloads using Azure Batch and Batch
 Shipyard.
 
-## Batch Shipyard Compute Node OS Support
+## Batch Shipyard Compute Node Host OS Support
 Batch Shipyard is currently compatible with most Azure Batch supported
 [Marketplace Linux VMs](https://docs.microsoft.com/azure/virtual-machines/linux/endorsed-distros),
 [compliant Linux custom images](http://batch-shipyard.readthedocs.io/en/latest/63-batch-shipyard-custom-images/),
@@ -128,7 +128,8 @@ and native Azure Batch
 [Windows Server with Containers](https://azuremarketplace.microsoft.com/marketplace/apps/Microsoft.WindowsServer?tab=Overview)
 VMs. Please see the
 [platform image support](http://batch-shipyard.readthedocs.io/en/latest/25-batch-shipyard-platform-image-support/)
-documentation for more information.
+documentation for more information specific to Batch Shipyard support of
+compute node host operating systems.
 
 ## Change Log
 Please see the

convoy/version.py

@@ -22,4 +22,4 @@
 # FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
 # DEALINGS IN THE SOFTWARE.
 
-__version__ = '3.2.0'
+__version__ = '3.3.0'

docs/20-batch-shipyard-usage.md

@@ -601,14 +601,17 @@ pool. This command may require a valid SSH user.
 * `nodes grls` will retrieve all of the remote login settings for every node
 in the specified pool
 * `nodes list` will list all nodes in the specified pool
-* `nodes prune` will prune unused Docker data
+* `nodes prune` will prune unused Docker data. This command requires a
+provisioned SSH user.
     * `--volumes` will also include volumes
-* `nodes ps` will list all Docker containers and their status
+* `nodes ps` will list all Docker containers and their status. This command
+requires a provisioned SSH user.
 * `nodes reboot` will reboot a specified node in the pool
     * `--all-start-task-failed` will reboot all nodes in the start task
       failed state
     * `--nodeid` is the node id to reboot
-* `nodes zap` will send a kill signal to all running Docker containers
+* `nodes zap` will send a kill signal to **all** running Docker containers.
+This command requires a provisioned SSH user.
     * `--no-remove` will not remove exited containers
     * `--stop` will execute docker stop instead
 * `rdp` will interactively log into a compute node via RDP. If neither

docs/80-batch-shipyard-multi-instance-tasks.md

@@ -233,10 +233,11 @@ Since Singularity is inherently easier to use in conjunction with MPI programs,
 it may be beneficial to use Singularity to containerize your MPI application
 instead of Docker. There is no need to package a launcher with your image
 and is handled more elegantly in Batch Shipyard without the need to split
-the execution and deal with potential cleanup artifacts. Additionally, there
-has been [some research](https://arxiv.org/abs/1709.10140) to indicate that
-Singularity outperforms other OS virtualization approaches and is more
-amenable for HPC applications.
+the execution and deal with potential cleanup artifacts if executing in
+non-`native` mode. Additionally, there has been
+[published research](https://arxiv.org/abs/1709.10140) to indicate that
+Singularity outperforms other OS virtualization approaches for certain
+benchmarks and is more amenable for HPC applications.
 
 ### More Information
 For more general information about MPI and Azure Batch, please visit

docs/97-faq.md

@@ -69,18 +69,21 @@ Advantages of `native` mode are:
 
 * Batch Shipyard with a provisioned SSH user is no longer necessary to
 perform actions such as terminating tasks where tasks are still running or
-deleting jobs with running tasks.
-* Multi-instance task execution is cleaner. You can execute multiple
-multi-instance tasks per job.
-* Potentially faster provisioning times, particularly pools with GPU devices.
+deleting jobs with running tasks in cases where normal Batch task termination
+fails to properly end the Docker container processes.
+* Multi-instance task execution (e.g., MPI job) is cleaner. You can execute
+multiple multi-instance tasks per job. If your workload is predominantly
+multi-instance, then it is recommended to use `native` mode.
 
 Disadvantages of `native` mode are:
 
 * Singularity containers are not supported.
-* `input_data` of any kind at the task-level is not possible.
-* `output_data` to `azure_storage` Azure Files (i.e., file shares) is not
-possible.
+* `input_data` of any kind at the task-level is not possible; you must either
+use `resource_files` or build your own solution.
+* `output_data` options are limited and egress to `azure_storage` Azure Files
+(i.e., file shares) is not possible.
 * Peer-to-peer distribution of Docker and Singularity images is not possible.
+* Other experimental features may not be supported.
 
 #### Does Batch Shipyard support Linux custom images?
 * Yes, please see [the guide](63-batch-shipyard-custom-images.md).