batch-shipyard/recipes/README.md

Batch Shipyard Recipes

This directory contains recipes and sample batch-style Docker workloads for use with Batch Shipyard on Azure Batch. Please note that all recipes have sample configurations that you can use to quickly get going, however, some of the config files cannot be used as-is as they may need to be modified for specific values (such as the credentials.yaml file which needs to be populated with your Batch and Storage account credentials). Please review each config file you are planning on using and modify accordingly. As the config samples are bare examples only, you will need to configure them to your liking for actual workloads.

If you would like to contribute your recipe, please take a look at this guide before submitting a pull request.

Recipe Collections

Use the following links to quickly navigate to recipe collections:

1. Benchmarks
2. Computational Fluid Dynamics (CFD)
3. Deep Learning
4. Molecular Dynamics (MD)
5. RemoteFS
6. Video Processing

Benchmarks

HPCG-Infiniband-IntelMPI

This HPCG-Infiniband-IntelMPI recipe contains information on how to Dockerize HPCG across Infiniband/RDMA Azure VMs with Intel MPI.

HPLinpack-Infiniband-IntelMPI

This HPLinpack-Infiniband-IntelMPI recipe contains information on how to Dockerize HPLinpack (HPL) across Infiniband/RDMA Azure VMs with Intel MPI.

Computational Fluid Dynamics (CFD)

OpenFOAM-Infiniband-IntelMPI

This OpenFOAM-Infiniband-IntelMPI recipe contains information on how to Dockerized distributed OpenFOAM across Infiniband/RDMA Azure VMs with Intel MPI.

OpenFOAM-TCP-OpenMPI

This OpenFOAM-TCP-OpenMPI recipe contains information on how to Dockerized distributed OpenFOAM across multiple Azure Batch compute nodes.

Deep Learning

CNTK-CPU-Infiniband-IntelMPI

This CNTK-CPU-Infiniband-IntelMPI recipe contains information on how to Dockerize CNTK for CPUs, including execution across multiple Infiniband/RDMA Azure VMs with multi-instance tasks.

CNTK-GPU-Infiniband-IntelMPI

This CNTK-GPU-Infiniband-IntelMPI recipe contains information on how to Dockerize CNTK on GPUs for use with N-Series Azure VMs, including execution across multiple Infiniband/RDMA Azure VMs with multi-instance tasks.

CNTK-CPU-OpenMPI

This CNTK-CPU-OpenMPI recipe contains information on how to Dockerize CNTK for CPUs, including execution across multiple compute nodes with multi-instance tasks.

CNTK-GPU-OpenMPI

This CNTK-GPU-OpenMPI recipe contains information on how to Dockerize CNTK on GPUs for use with N-Series Azure VMs, including execution across multiple compute nodes and multiple GPUs with multi-instance tasks.

Caffe-CPU

This Caffe-CPU recipe contains information on how to Dockerize Caffe for use on Azure Batch compute nodes.

Caffe-GPU

This Caffe-GPU recipe contains information on how to Dockerize Caffe on GPUs for use with N-Series Azure VMs.

Caffe2-CPU

This Caffe2-CPU recipe contains information on how to Dockerize Caffe2 for use on Azure Batch compute nodes.

Caffe2-GPU

This Caffe2-GPU recipe contains information on how to Dockerize Caffe2 on GPUs for use with N-Series Azure VMs.

Chainer-CPU

This Chainer-CPU recipe contains information on how to Dockerize Chainer for use on Azure Batch compute nodes.

Chainer-GPU

This Chainer-GPU recipe contains information on how to Dockerize Chainer on GPUs for use with N-Series Azure VMs.

Keras+Theano-CPU

This Keras+Theano-CPU recipe contains information on how to Dockerize Keras with the Theano backend for use on Azure Batch compute nodes.

Keras+Theano-GPU

This Keras+Theano-GPU recipe contains information on how to Dockerize Keras with the Theano backend for use with N-Series Azure VMs.

MXNet-CPU

This MXNet-CPU recipe contains information on how to Dockerize MXNet for use on Azure Batch compute nodes, including execution across multiple compute nodes with multi-instance tasks.

MXNet-GPU

This MXNet-GPU recipe contains information on how to Dockerize MXNet on GPUs for use with N-Series Azure VMs, including execution across multiple compute nodes and multiple GPUs with multi-instance tasks.

TensorFlow-CPU

This TensorFlow-CPU recipe contains information on how to Dockerize TensorFlow for use on Azure Batch compute nodes.

TensorFlow-Distributed

This TensorFlow-Distributed recipe contains information on how to Dockerize TensorFlow on GPUs for use with N-series Azure VMs or across multiple CPU nodes.

TensorFlow-GPU

This TensorFlow-GPU recipe contains information on how to Dockerize TensorFlow on GPUs for use with N-series Azure VMs.

Torch-CPU

This Torch-CPU recipe contains information on how to Dockerize Torch for use on Azure Batch compute nodes.

Torch-GPU

This Torch-GPU recipe contains information on how to Dockerize Torch on GPUs for use with N-series Azure VMs.

Molecular Dynamics (MD)

NAMD-GPU

This NAMD-GPU recipe contains information on how to Dockerize NAMD on GPUs for use with N-Series Azure VMs.

NAMD-Infiniband-IntelMPI

This NAMD-Infiniband-IntelMPI recipe contains information on how to Dockerize distributed NAMD across Infiniband/RDMA Azure VMs with Intel MPI.

NAMD-TCP

This NAMD-TCP recipe contains information on how to Dockerize distributed NAMD across multiple Azure Batch compute nodes using TCP.

RemoteFS Provisioning

RemoteFS-GlusterFS

This RemoteFS-GlusterFS recipe contains information on how to provision a sample multi-VM GlusterFS storage cluster.

RemoteFS-GlusterFS+BatchPool

This RemoteFS-GlusterFS+BatchPool recipe contains information on how to provision a Batch pool and automatically link it against a provisioned GlusterFS storage cluster.

RemoteFS-NFS

This RemoteFS-NFS recipe contains information on how to provision a sample single VM NFS server.

Video Processing

FFmpeg-GPU

This recipe contains information on how to use Dockerized FFmpeg on GPUs for use with the N-series Azure VMs.