batch-shipyard/recipes

Batch Shipyard Recipes

Batch Shipyard can accommodate most containerized Batch and HPC workloads. This area contains recipes and sample container workloads that may relate to your own anticipated scenario. Batch Shipyard is not limited to these programs or types of workloads represented by these recipes, but can run arbitrary containers to execute your Batch and HPC scenario.

Please note that all recipes have sample configurations that you can use to quickly get started, however, some of the config files cannot be used as-is as they may need to be modified for specific values (such as the credentials.yaml file which needs to be populated with your Batch and Storage account credentials). Please review each config file you are planning on using and modify accordingly. As these recipe configuration are intended to show example usage, you will need to modify and configure them for your actual workloads.

If you would like to contribute your recipe, please take a look at this guide before submitting a pull request.

Recipe Collections

Use the following links to quickly navigate to recipe collections:

1. Benchmarks
2. Computational Fluid Dynamics (CFD)
3. Deep Learning
4. Genomics and Bioinformatics
5. Molecular Dynamics (MD)
6. RemoteFS
7. Slurm on Batch
8. Video Processing
9. Windows

Benchmarks

HPCG-Infiniband-IntelMPI

This HPCG-Infiniband-IntelMPI recipe contains information on how to containerize HPCG across Infiniband/RDMA Azure VMs with Intel MPI.

HPLinpack-Infiniband-IntelMPI

This HPLinpack-Infiniband-IntelMPI recipe contains information on how to containerize HPLinpack (HPL) across Infiniband/RDMA Azure VMs with Intel MPI.

mpiBench-Infiniband-MPICH

This mpiBench-Infinibnad-MPICH recipe contains information on how to containerize mpiBench across Infiniband/RDMA Azure VMs with MPICH.

mpiBench-Infiniband-OpenMPI

This mpiBench-Infinibnad-MPICH recipe contains information on how to containerize mpiBench across Infiniband/RDMA Azure VMs with MPICH.

mpiBench-IntelMPI

This mpiBench-IntelMPI recipe contains information on how to containerize mpiBench across Azure VMs with Intel MPI.

mpiBench-MPICH

This mpiBench-MPICH recipe contains information on how to containerize mpiBench across Azure VMs with MPICH.

mpiBench-OpenMPI

This mpiBench-OpenMPI recipe contains information on how to containerize mpiBench across Azure VMs with Open MPI.

OSUMicroBenchmarks-Infiniband-MVAPICH

This OSUMicroBenchmarks-Infiniband-MVAPICH recipe contains information on how to containers the OSU Micro-Benchmarks across Infiniband/RDMA Azure VMs with MVAPICH.

Computational Fluid Dynamics (CFD)

OpenFOAM-Infiniband-IntelMPI

This OpenFOAM-Infiniband-IntelMPI recipe contains information on how to containerize distributed OpenFOAM across Infiniband/RDMA Azure VMs with Intel MPI.

OpenFOAM-TCP-OpenMPI

This OpenFOAM-TCP-OpenMPI recipe contains information on how to containerize distributed OpenFOAM across multiple Azure Batch compute nodes.

Deep Learning

Caffe-CPU

This Caffe-CPU recipe contains information on how to containerize Caffe for use on Azure Batch compute nodes.

Caffe-GPU

This Caffe-GPU recipe contains information on how to containerize Caffe on GPUs for use with N-Series Azure VMs.

Caffe2-CPU

This Caffe2-CPU recipe contains information on how to containerize Caffe2 for use on Azure Batch compute nodes.

Caffe2-GPU

This Caffe2-GPU recipe contains information on how to containerize Caffe2 on GPUs for use with N-Series Azure VMs.

Chainer-CPU

This Chainer-CPU recipe contains information on how to containerize Chainer for use on Azure Batch compute nodes.

Chainer-GPU

This Chainer-GPU recipe contains information on how to containerize Chainer on GPUs for use with N-Series Azure VMs.

CNTK-CPU-Infiniband-IntelMPI

This CNTK-CPU-Infiniband-IntelMPI recipe contains information on how to containerize CNTK for CPUs, including execution across multiple Infiniband/RDMA Azure VMs with multi-instance tasks.

CNTK-CPU-OpenMPI

This CNTK-CPU-OpenMPI recipe contains information on how to containerize CNTK for CPUs, including execution across multiple compute nodes with multi-instance tasks.

CNTK-GPU-Infiniband-IntelMPI

This CNTK-GPU-Infiniband-IntelMPI recipe contains information on how to containerize CNTK on GPUs for use with N-Series Azure VMs, including execution across multiple Infiniband/RDMA Azure VMs with multi-instance tasks.

CNTK-GPU-OpenMPI

This CNTK-GPU-OpenMPI recipe contains information on how to containerize CNTK on GPUs for use with N-Series Azure VMs, including execution across multiple compute nodes and multiple GPUs with multi-instance tasks.

HPMLA-CPU-OpenMPI

This recipe contains information on how to containerize the Microsoft High Performance ML Algorithms (HPMLA) for use across multiple compute nodes.

Keras+Theano-CPU

This Keras+Theano-CPU recipe contains information on how to containerize Keras with the Theano backend for use on Azure Batch compute nodes.

Keras+Theano-GPU

This Keras+Theano-GPU recipe contains information on how to containerize Keras with the Theano backend for use with N-Series Azure VMs.

MXNet-CPU

This MXNet-CPU recipe contains information on how to containerize MXNet for use on Azure Batch compute nodes, including execution across multiple compute nodes with multi-instance tasks.

MXNet-GPU

This MXNet-GPU recipe contains information on how to containerize MXNet on GPUs for use with N-Series Azure VMs, including execution across multiple compute nodes and multiple GPUs with multi-instance tasks.

PyTorch-CPU

This PyTorch-CPU recipe contains information on how to containerize PyTorch for use on Azure Batch compute nodes.

PyTorch-GPU

This PyTorch-GPU recipe contains information on how to containerize PyTorch on GPUs for use with N-series Azure VMs.

TensorFlow-CPU

This TensorFlow-CPU recipe contains information on how to containerize TensorFlow for use on Azure Batch compute nodes.

TensorFlow-Distributed

This TensorFlow-Distributed recipe contains information on how to containerize TensorFlow on GPUs for use with N-series Azure VMs or across multiple CPU nodes.

TensorFlow-GPU

This TensorFlow-GPU recipe contains information on how to containerize TensorFlow on GPUs for use with N-series Azure VMs.

Torch-CPU

This Torch-CPU recipe contains information on how to containerize Torch for use on Azure Batch compute nodes.

Torch-GPU

This Torch-GPU recipe contains information on how to containerize Torch on GPUs for use with N-series Azure VMs.

Genomics and Bioinformatics

BLAST-CPU

This BLAST-CPU recipes contains information on how to containerize a BLAST pipeline with parallelization using Azure Batch compute nodes.

RNASeq-CPU

This RNASeq-CPU recipes contains information on how to containerize a proof-of-concept RNA-Seq pipeline using Azure Batch compute nodes.

Molecular Dynamics (MD)

NAMD-GPU

This NAMD-GPU recipe contains information on how to containerize NAMD on GPUs for use with N-Series Azure VMs.

NAMD-Infiniband-IntelMPI

This NAMD-Infiniband-IntelMPI recipe contains information on how to containerize distributed NAMD across Infiniband/RDMA Azure VMs with Intel MPI.

NAMD-TCP

This NAMD-TCP recipe contains information on how to containerize distributed NAMD across multiple Azure Batch compute nodes using TCP.

RemoteFS Provisioning

RemoteFS-GlusterFS

This RemoteFS-GlusterFS recipe contains information on how to provision a sample multi-VM GlusterFS storage cluster.

RemoteFS-GlusterFS+BatchPool

This RemoteFS-GlusterFS+BatchPool recipe contains information on how to provision a Batch pool and automatically link it against a provisioned GlusterFS storage cluster.

RemoteFS-NFS

This RemoteFS-NFS recipe contains information on how to provision a sample single VM NFS server.

Slurm on Batch

Slurm+NFS

This recipe contains information on how to orchestrate a Slurm cluster with an NFS shared file system.

Video Processing

FFmpeg-GPU

This recipe contains information on how to containerize FFmpeg on GPUs for use with the N-series Azure VMs.

Windows

DiskSpd-Windows

This recipe contains information on how to run a containerized version of the DiskSpd tool with Windows Server Containers on Azure VMs.

DotNet-Windows

This recipe contains information on how to run a sample DotNet application with Windows Server Containers on Azure VMs.