457ca75e6f
|
||
|---|---|---|
| cascade | ||
| config_templates | ||
| docs | ||
| resources | ||
| scripts | ||
| .gitignore | ||
| .travis.yml | ||
| CHANGELOG.md | ||
| CONTRIBUTING.md | ||
| Dockerfile | ||
| LICENSE | ||
| README.md | ||
| requirements.txt | ||
| shipyard.py | ||
README.md
Batch Shipyard
Batch Shipyard is a tool to help provision and execute Dockerized workloads on Azure Batch compute pools. No experience with the Azure Batch SDK is needed; run your batch-style Docker tasks through easy-to-understand configuration files!
Major Features
- Automated Docker Host Engine installation on compute nodes
- Automated deployment of required Docker images to compute nodes
- Accelerated Docker image deployment at scale to compute pools consisting of a large number of compute nodes via peer-to-peer distribution of Docker images among the VMs
- Automated Docker Private Registry instance creation on compute nodes backed to Azure Storage if specified
- Automated Azure File Docker Volume Driver installation and share setup for SMB/CIFS backed to Azure Storage if specified
- Seamless integration with Azure Batch job, task and file concepts along with full pass-through of the Azure Batch API to containers executed on compute nodes
- Support for task dependencies to allow for complex processing pipelines with Docker containers
- Support for multi-instance tasks to accomodate Dockerized MPI applications on compute pools
- Transparent assist for creating Docker containers utilizing Infiniband/RDMA for MPI on HPC low-latency Azure VM instances (STANDARD_A8 and STANDARD_A9)
- Automatic set up of SSH tunneling to Docker Hosts on compute nodes if specified
Installation
Simply clone the repository:
git clone https://github.com/Azure/batch-shipyard.git
or download the latest release.
Requirements
The Batch Shipyard tool is written in Python. The client script is compatible
with Python 2.7 or 3.3+. You will also need to install the
Azure Batch and
Azure Storage python packages.
Installation can be performed using the requirements.txt
file via the command pip install --user -r requirements.txt (or via pip3
for python3).
Host OS (Compute Node) Support
Batch Shipyard is currently only compatible with Linux Batch Compute Pools configured via VirtualMachineConfiguration. Please see the list of Azure Batch supported Marketplace Linux VMs for use with Batch Shipyard.
Documentation
Please refer to this guide for a complete primer on concepts and usage.
Limitations
- Oracle Linux is not supported with Batch Shipyard at this time.
- Task dependencies are incompatible with multi-instance tasks. This is a current limitation of the underlying Azure Batch service.
- Only Intel MPI can be used in conjunction Infiniband/RDMA on Azure Linux VMs. This is a current limitation of the underlying VM and host drivers.
ChangeLog
See the CHANGELOG.md file.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.