From e00f1e64c079d7d43efa34941bd408e2a09792c4 Mon Sep 17 00:00:00 2001 From: Fred Park Date: Thu, 1 Mar 2018 08:25:49 -0800 Subject: [PATCH] Tag for 3.3.0 release --- CHANGELOG.md | 56 ++++++++++++++----- README.md | 39 ++++++------- convoy/version.py | 2 +- docs/20-batch-shipyard-usage.md | 9 ++- .../80-batch-shipyard-multi-instance-tasks.md | 9 +-- docs/97-faq.md | 17 +++--- 6 files changed, 83 insertions(+), 49 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dccf274..8793b8c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,21 +2,44 @@ ## [Unreleased] +## [3.3.0] - 2018-03-01 ### Added -- `storage sas create` command added as a utility helper function to create -SAS tokens for given storage accounts in credentials +- Support for specifying default task exit conditions (i.e., non-zero exit +codes). Please see the jobs configuration doc for more information. +- New commands (please see usage doc for more information): + - `pool nodes prune` will remove all unused Docker-related data on all + nodes in the pool (requires an SSH user) + - `pool nodes ps` performs a `docker ps -a` on all nodes in the pool + (requires an SSH user) + - `pool nodes zap` will kill (and optionally remove) **all** running + Docker containers on nodes in a pool (requires an SSH user). Note that + `jobs tasks term` is the preferred command to control individual + (or grouped) task termination or `jobs term --termtasks` to terminate + at the job level. + - `storage sas create` command added as a utility helper function to + create SAS tokens for given storage accounts in credentials - Support for activating [Azure Hybrid Use Benefit](https://azure.microsoft.com/pricing/hybrid-benefit/) for Windows pools ### Changed -- Expand node error details +- Greatly expand pool, node, job, and task details for `list` sub-commands +- Expand error detail key/value pairs if present - Name resources for RemoteFS with 3 digits (e.g., 000 instead of 0) for improved alpha ordering +- Move site extension to nuget.org (#172) ### Fixed -- Remote FS cluster del with disks error +- Command lines in non-native mode now properly expand environment variables +with normal quoting +- RemoteFS cluster del with disks error +- RemoteFS cluster status decoding issues - Unintended interaction between native mode and custom images +- Handle starting Docker service for all deployments +- Do not automatically mount storage cluster or custom linux mounts on boot +due to potential race conditions with the ephemeral disk mount +- Fix TLS issue with powershell (#171) +- Fix conflicts when using install.sh with Anaconda environments - Update packer scripts and fix typos - Minor doc updates @@ -24,18 +47,20 @@ improved alpha ordering ### Added - Custom Linux Mount support for `shared_data_volumes`. Please see the global configuration doc for more information. -- `account` command added with the following sub-commands (requires AAD auth): - - `info` provides information about a Batch account (including account - level quotas) - - `list` provides information about all (or a resource group subset) of - accounts within the subscription specified in credentials - - `quota` provides service level quota information for the subscription - for a given location +- New commands (please see usage doc for more information): + - `account` command added with the following sub-commands (requires + AAD auth): + - `info` provides information about a Batch account (including account + level quotas) + - `list` provides information about all (or a resource group subset) + of accounts within the subscription specified in credentials + - `quota` provides service level quota information for the + subscription for a given location + - `pool rdp` sub-command added, please see usage doc for more information. + Requires Batch Shipyard executing on Windows with target Windows + containers pools. - `pool images update` command now supports updating Docker images in native container support pools via SSH -- `pool rdp` sub-command added, please see usage doc for more information. -Requires Batch Shipyard executing on Windows with target Windows containers -pools. - Ability to specify an AAD authority URL via the `aad`:`authority_url` credential configuration, `--aad-authority-url` command line option or `SHIPYARD_AAD_AUTHORITY_URL` environment variable. Please see relevant @@ -1139,7 +1164,8 @@ transfer is disabled #### Added - Initial release -[Unreleased]: https://github.com/Azure/batch-shipyard/compare/3.2.0...HEAD +[Unreleased]: https://github.com/Azure/batch-shipyard/compare/3.3.0...HEAD +[3.3.0]: https://github.com/Azure/batch-shipyard/compare/3.2.0...3.3.0 [3.2.0]: https://github.com/Azure/batch-shipyard/compare/3.1.0...3.2.0 [3.1.0]: https://github.com/Azure/batch-shipyard/compare/3.0.3...3.1.0 [3.0.3]: https://github.com/Azure/batch-shipyard/compare/3.0.2...3.0.3 diff --git a/README.md b/README.md index ef9f001..13461c2 100644 --- a/README.md +++ b/README.md @@ -17,13 +17,6 @@ Additionally, Batch Shipyard provides the ability to provision and manage entire [standalone remote file systems (storage clusters)](http://batch-shipyard.readthedocs.io/en/latest/65-batch-shipyard-remote-fs/) in Azure, independent of any integrated Azure Batch functionality. -Batch Shipyard is now integrated directly into -[Azure Cloud Shell](https://docs.microsoft.com/azure/cloud-shell/overview) -and you can execute any Batch Shipyard workload using your web browser or -the Microsoft Azure [Android](https://play.google.com/store/apps/details?id=com.microsoft.azure&hl=en) -and [iOS](https://itunes.apple.com/us/app/microsoft-azure/id1219013620?mt=8) -app. - ## Major Features * Automated [Docker Host Engine](https://www.docker.com) and [Singularity](http://singularity.lbl.gov/) installations tuned for @@ -79,10 +72,10 @@ on [Azure N-Series VM instances](https://docs.microsoft.com/azure/virtual-machin applications packaged in Docker or Singularity on compute pools with automatic job completion and task termination * Transparent assist for running Docker and Singularity containers utilizing -Infiniband/RDMA for MPI on HPC low-latency Azure VM instances: - * [A-Series](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-hpc): STANDARD\_A8, STANDARD\_A9 - * [H-Series](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-hpc): STANDARD\_H16R, STANDARD\_H16MR - * [N-Series](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-gpu): STANDARD\_NC24R, STANDARD\_NC24RS\_V2, STANDARD\_ND24RS +Infiniband/RDMA for MPI on HPC low-latency Azure VM instances including +[A-Series](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-hpc), +[H-Series](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-hpc), +and [N-Series](https://docs.microsoft.com/azure/virtual-machines/linux/sizes-gpu) * Support for [Azure Batch task dependencies](https://azure.microsoft.com/documentation/articles/batch-task-dependencies/) allowing complex processing pipelines and DAGs * Support for merge or final task specification that automatically depends @@ -96,20 +89,27 @@ optional creation of SSH tunneling scripts to Docker Hosts on compute nodes [Azure KeyVault](https://azure.microsoft.com/services/key-vault/) * Support for [custom host images](http://batch-shipyard.readthedocs.io/en/latest/63-batch-shipyard-custom-images/) * Support for [Windows Containers](https://docs.microsoft.com/virtualization/windowscontainers/about/) -on compliant Windows compute node pools +on compliant Windows compute node pools with the ability to activate +[Azure Hybrid Use Benefit](https://azure.microsoft.com/pricing/hybrid-benefit/) +if applicable ## Installation ### Azure Cloud Shell -Batch Shipyard is now integrated into Azure Cloud Shell with no installation -required. Simply request a Cloud Shell session and type `shipyard` to invoke -the CLI. +Batch Shipyard is integrated directly into +[Azure Cloud Shell](https://docs.microsoft.com/azure/cloud-shell/overview) +and you can execute any Batch Shipyard workload using your web browser or +the Microsoft Azure [Android](https://play.google.com/store/apps/details?id=com.microsoft.azure&hl=en) +and [iOS](https://itunes.apple.com/us/app/microsoft-azure/id1219013620?mt=8) +app. -Try Batch Shipyard now from your browser: +Simply request a Cloud Shell session and type `shipyard` to invoke the CLI; +no installation is required. Try Batch Shipyard now from your browser: [![Launch Cloud Shell](https://shell.azure.com/images/launchcloudshell.png "Launch Cloud Shell")](https://shell.azure.com) ### Local Installation Please see [the installation guide](http://batch-shipyard.readthedocs.io/en/latest/01-batch-shipyard-installation/) -for more information regarding installation and requirements. +for more information regarding the various local installation options and +requirements. ## Documentation and Recipes Please refer to the @@ -120,7 +120,7 @@ Visit the section for various sample container workloads using Azure Batch and Batch Shipyard. -## Batch Shipyard Compute Node OS Support +## Batch Shipyard Compute Node Host OS Support Batch Shipyard is currently compatible with most Azure Batch supported [Marketplace Linux VMs](https://docs.microsoft.com/azure/virtual-machines/linux/endorsed-distros), [compliant Linux custom images](http://batch-shipyard.readthedocs.io/en/latest/63-batch-shipyard-custom-images/), @@ -128,7 +128,8 @@ and native Azure Batch [Windows Server with Containers](https://azuremarketplace.microsoft.com/marketplace/apps/Microsoft.WindowsServer?tab=Overview) VMs. Please see the [platform image support](http://batch-shipyard.readthedocs.io/en/latest/25-batch-shipyard-platform-image-support/) -documentation for more information. +documentation for more information specific to Batch Shipyard support of +compute node host operating systems. ## Change Log Please see the diff --git a/convoy/version.py b/convoy/version.py index 628199f..73038b0 100644 --- a/convoy/version.py +++ b/convoy/version.py @@ -22,4 +22,4 @@ # FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER # DEALINGS IN THE SOFTWARE. -__version__ = '3.2.0' +__version__ = '3.3.0' diff --git a/docs/20-batch-shipyard-usage.md b/docs/20-batch-shipyard-usage.md index 33cab8c..79f642a 100644 --- a/docs/20-batch-shipyard-usage.md +++ b/docs/20-batch-shipyard-usage.md @@ -601,14 +601,17 @@ pool. This command may require a valid SSH user. * `nodes grls` will retrieve all of the remote login settings for every node in the specified pool * `nodes list` will list all nodes in the specified pool -* `nodes prune` will prune unused Docker data +* `nodes prune` will prune unused Docker data. This command requires a +provisioned SSH user. * `--volumes` will also include volumes -* `nodes ps` will list all Docker containers and their status +* `nodes ps` will list all Docker containers and their status. This command +requires a provisioned SSH user. * `nodes reboot` will reboot a specified node in the pool * `--all-start-task-failed` will reboot all nodes in the start task failed state * `--nodeid` is the node id to reboot -* `nodes zap` will send a kill signal to all running Docker containers +* `nodes zap` will send a kill signal to **all** running Docker containers. +This command requires a provisioned SSH user. * `--no-remove` will not remove exited containers * `--stop` will execute docker stop instead * `rdp` will interactively log into a compute node via RDP. If neither diff --git a/docs/80-batch-shipyard-multi-instance-tasks.md b/docs/80-batch-shipyard-multi-instance-tasks.md index 7b050a4..b3ff4cc 100644 --- a/docs/80-batch-shipyard-multi-instance-tasks.md +++ b/docs/80-batch-shipyard-multi-instance-tasks.md @@ -233,10 +233,11 @@ Since Singularity is inherently easier to use in conjunction with MPI programs, it may be beneficial to use Singularity to containerize your MPI application instead of Docker. There is no need to package a launcher with your image and is handled more elegantly in Batch Shipyard without the need to split -the execution and deal with potential cleanup artifacts. Additionally, there -has been [some research](https://arxiv.org/abs/1709.10140) to indicate that -Singularity outperforms other OS virtualization approaches and is more -amenable for HPC applications. +the execution and deal with potential cleanup artifacts if executing in +non-`native` mode. Additionally, there has been +[published research](https://arxiv.org/abs/1709.10140) to indicate that +Singularity outperforms other OS virtualization approaches for certain +benchmarks and is more amenable for HPC applications. ### More Information For more general information about MPI and Azure Batch, please visit diff --git a/docs/97-faq.md b/docs/97-faq.md index 764432c..e77c1e1 100644 --- a/docs/97-faq.md +++ b/docs/97-faq.md @@ -69,18 +69,21 @@ Advantages of `native` mode are: * Batch Shipyard with a provisioned SSH user is no longer necessary to perform actions such as terminating tasks where tasks are still running or -deleting jobs with running tasks. -* Multi-instance task execution is cleaner. You can execute multiple -multi-instance tasks per job. -* Potentially faster provisioning times, particularly pools with GPU devices. +deleting jobs with running tasks in cases where normal Batch task termination +fails to properly end the Docker container processes. +* Multi-instance task execution (e.g., MPI job) is cleaner. You can execute +multiple multi-instance tasks per job. If your workload is predominantly +multi-instance, then it is recommended to use `native` mode. Disadvantages of `native` mode are: * Singularity containers are not supported. -* `input_data` of any kind at the task-level is not possible. -* `output_data` to `azure_storage` Azure Files (i.e., file shares) is not -possible. +* `input_data` of any kind at the task-level is not possible; you must either +use `resource_files` or build your own solution. +* `output_data` options are limited and egress to `azure_storage` Azure Files +(i.e., file shares) is not possible. * Peer-to-peer distribution of Docker and Singularity images is not possible. +* Other experimental features may not be supported. #### Does Batch Shipyard support Linux custom images? * Yes, please see [the guide](63-batch-shipyard-custom-images.md).