e2390cd3f1
- Resolves #139 |
||
|---|---|---|
| cargo | ||
| cascade | ||
| config_templates | ||
| contrib | ||
| convoy | ||
| docker | ||
| docs | ||
| recipes | ||
| scripts | ||
| site-extension | ||
| .gitattributes | ||
| .gitignore | ||
| .travis.yml | ||
| CHANGELOG.md | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| LICENSE | ||
| README.md | ||
| THIRD_PARTY_NOTICES.txt | ||
| appveyor.yml | ||
| install.cmd | ||
| install.sh | ||
| mkdocs.yml | ||
| requirements.txt | ||
| shipyard.py | ||
README.md
Batch Shipyard
Batch Shipyard is a tool to help provision and execute container-based batch processing and HPC workloads on Azure Batch compute pools. Batch Shipyard supports both Docker and Singularity containers! No experience with the Azure Batch SDK is needed; run your containers with easy-to-understand configuration files.
Additionally, Batch Shipyard provides the ability to provision and manage entire standalone remote file systems (storage clusters) in Azure, independent of any integrated Azure Batch functionality.
Batch Shipyard is now integrated directly into Azure Cloud Shell and you can execute any Batch Shipyard workload using your web browser or the Microsoft Azure Android and iOS app.
Major Features
- Automated Docker Host Engine and Singularity installations tuned for Azure Batch compute nodes
- Automated deployment of required Docker and/or Singularity images to compute nodes
- Accelerated Docker and Singularity image deployment at scale to compute pools consisting of a large number of VMs via private peer-to-peer distribution of container images among the compute nodes
- Mixed mode support for Docker and Singularity: run your Docker and Singularity containers within the same job, side-by-side or even concurrently
- Comprehensive data movement support: move data easily between locally accessible storage systems, remote filesystems, Azure Blob or File Storage, and compute nodes
- Support for Docker Registries including Azure Container Registry and other Internet-accessible public and private registries
- Support for the Singularity Hub Container Registry
- Standalone Remote Filesystem Provisioning with integration to auto-link these filesystems to compute nodes with support for
- Automatic shared data volume support
- Remote Filesystems as provisioned by Batch Shipyard
- Azure File via SMB
- GlusterFS provisioned directly on compute nodes
- Seamless integration with Azure Batch job, task and file concepts along with full pass-through of the Azure Batch API to containers executed on compute nodes
- Support for Low Priority Compute Nodes
- Support for pool autoscale and autopool to dynamically scale and control computing resources on-demand
- Support for Task Factories with the ability to generate tasks based on parametric (parameter) sweeps, randomized input, file enumeration, replication, and custom Python code-based generators
- Support for deploying Batch compute nodes into a specified Virtual Network
- Transparent support for GPU-accelerated container applications on both Docker and Singularity on Azure N-Series VM instances
- Support for multi-instance tasks to accommodate MPI and multi-node cluster applications packaged in Docker or Singularity on compute pools with automatic job completion and task termination
- Transparent assist for running Docker and Singularity containers utilizing Infiniband/RDMA for MPI on HPC low-latency Azure VM instances:
- Support for Azure Batch task dependencies allowing complex processing pipelines and DAGs with containers
- Support for job schedules and recurrences for automatic execution of tasks at set intervals
- Support for live job and job schedule migration between pools
- Automatic setup of SSH users to all nodes in the compute pool and optional tunneling to Docker Hosts on compute nodes
- Support for credential management through Azure KeyVault
- Support for execution on an Azure Function App environment
- Support for custom host images
Installation
Azure Cloud Shell
Batch Shipyard is now integrated into Azure Cloud Shell with no installation
required. Simply request a Cloud Shell session and type shipyard to invoke
the CLI.
Local Installation
Installation is typically an easy two-step process. The CLI is also available as a Docker image: alfpark/batch-shipyard:latest-cli. Please see the installation guide for more information regarding installation and requirements.
Documentation and Recipes
Please refer to the Batch Shipyard Documentation on Read the Docs.
Visit the Batch Shipyard Recipes section for various sample Docker workloads using Azure Batch and Batch Shipyard.
Batch Shipyard Compute Node OS Support
Batch Shipyard is currently compatible with Azure Batch supported Marketplace Linux VMs and compliant Linux custom images.
Change Log
Please see the Change Log for project history.
Please see this project's Code of Conduct and Contributing guidelines.