11 KiB
Batch Shipyard Recipes
Batch Shipyard can accommodate most containerized Batch and HPC workloads. This area contains recipes and sample container workloads that may relate to your own anticipated scenario. Batch Shipyard is not limited to these programs or types of workloads represented by these recipes, but can run arbitrary containers to execute your Batch and HPC scenario.
Please note that all recipes have sample configurations that you can use to
quickly get started, however, some of the config files cannot be used as-is
as they may need to be modified for specific values (such as the
credentials.yaml file which needs to be populated with your Batch and
Storage account credentials). Please review each config file you are planning
on using and modify accordingly. As these recipe configuration are intended
to show example usage, you will need to modify and configure them for your
actual workloads.
If you would like to contribute your recipe, please take a look at this guide before submitting a pull request.
Recipe Collections
Use the following links to quickly navigate to recipe collections:
- Benchmarks
- Computational Fluid Dynamics (CFD)
- Deep Learning
- Genomics and Bioinformatics
- Molecular Dynamics (MD)
- RemoteFS
- Slurm on Batch
- Video Processing
- Windows
Benchmarks
HPCG-Infiniband-IntelMPI
This HPCG-Infiniband-IntelMPI recipe contains information on how to containerize HPCG across Infiniband/RDMA Azure VMs with Intel MPI.
HPLinpack-Infiniband-IntelMPI
This HPLinpack-Infiniband-IntelMPI recipe contains information on how to containerize HPLinpack (HPL) across Infiniband/RDMA Azure VMs with Intel MPI.
mpiBench-Infiniband-MPICH
This mpiBench-Infinibnad-MPICH recipe contains information on how to containerize mpiBench across Infiniband/RDMA Azure VMs with MPICH.
mpiBench-Infiniband-OpenMPI
This mpiBench-Infinibnad-MPICH recipe contains information on how to containerize mpiBench across Infiniband/RDMA Azure VMs with MPICH.
mpiBench-IntelMPI
This mpiBench-IntelMPI recipe contains information on how to containerize mpiBench across Azure VMs with Intel MPI.
mpiBench-MPICH
This mpiBench-MPICH recipe contains information on how to containerize mpiBench across Azure VMs with MPICH.
mpiBench-OpenMPI
This mpiBench-OpenMPI recipe contains information on how to containerize mpiBench across Azure VMs with Open MPI.
OSUMicroBenchmarks-Infiniband-MVAPICH
This OSUMicroBenchmarks-Infiniband-MVAPICH recipe contains information on how to containers the OSU Micro-Benchmarks across Infiniband/RDMA Azure VMs with MVAPICH.
Computational Fluid Dynamics (CFD)
OpenFOAM-Infiniband-IntelMPI
This OpenFOAM-Infiniband-IntelMPI recipe contains information on how to containerize distributed OpenFOAM across Infiniband/RDMA Azure VMs with Intel MPI.
OpenFOAM-TCP-OpenMPI
This OpenFOAM-TCP-OpenMPI recipe contains information on how to containerize distributed OpenFOAM across multiple Azure Batch compute nodes.
Deep Learning
Caffe-CPU
This Caffe-CPU recipe contains information on how to containerize Caffe for use on Azure Batch compute nodes.
Caffe-GPU
This Caffe-GPU recipe contains information on how to containerize Caffe on GPUs for use with N-Series Azure VMs.
Caffe2-CPU
This Caffe2-CPU recipe contains information on how to containerize Caffe2 for use on Azure Batch compute nodes.
Caffe2-GPU
This Caffe2-GPU recipe contains information on how to containerize Caffe2 on GPUs for use with N-Series Azure VMs.
Chainer-CPU
This Chainer-CPU recipe contains information on how to containerize Chainer for use on Azure Batch compute nodes.
Chainer-GPU
This Chainer-GPU recipe contains information on how to containerize Chainer on GPUs for use with N-Series Azure VMs.
CNTK-CPU-Infiniband-IntelMPI
This CNTK-CPU-Infiniband-IntelMPI recipe contains information on how to containerize CNTK for CPUs, including execution across multiple Infiniband/RDMA Azure VMs with multi-instance tasks.
CNTK-CPU-OpenMPI
This CNTK-CPU-OpenMPI recipe contains information on how to containerize CNTK for CPUs, including execution across multiple compute nodes with multi-instance tasks.
CNTK-GPU-Infiniband-IntelMPI
This CNTK-GPU-Infiniband-IntelMPI recipe contains information on how to containerize CNTK on GPUs for use with N-Series Azure VMs, including execution across multiple Infiniband/RDMA Azure VMs with multi-instance tasks.
CNTK-GPU-OpenMPI
This CNTK-GPU-OpenMPI recipe contains information on how to containerize CNTK on GPUs for use with N-Series Azure VMs, including execution across multiple compute nodes and multiple GPUs with multi-instance tasks.
HPMLA-CPU-OpenMPI
This recipe contains information on how to containerize the Microsoft High Performance ML Algorithms (HPMLA) for use across multiple compute nodes.
Keras+Theano-CPU
This Keras+Theano-CPU recipe contains information on how to containerize Keras with the Theano backend for use on Azure Batch compute nodes.
Keras+Theano-GPU
This Keras+Theano-GPU recipe contains information on how to containerize Keras with the Theano backend for use with N-Series Azure VMs.
MXNet-CPU
This MXNet-CPU recipe contains information on how to containerize MXNet for use on Azure Batch compute nodes, including execution across multiple compute nodes with multi-instance tasks.
MXNet-GPU
This MXNet-GPU recipe contains information on how to containerize MXNet on GPUs for use with N-Series Azure VMs, including execution across multiple compute nodes and multiple GPUs with multi-instance tasks.
PyTorch-CPU
This PyTorch-CPU recipe contains information on how to containerize PyTorch for use on Azure Batch compute nodes.
PyTorch-GPU
This PyTorch-GPU recipe contains information on how to containerize PyTorch on GPUs for use with N-series Azure VMs.
TensorFlow-CPU
This TensorFlow-CPU recipe contains information on how to containerize TensorFlow for use on Azure Batch compute nodes.
TensorFlow-Distributed
This TensorFlow-Distributed recipe contains information on how to containerize TensorFlow on GPUs for use with N-series Azure VMs or across multiple CPU nodes.
TensorFlow-GPU
This TensorFlow-GPU recipe contains information on how to containerize TensorFlow on GPUs for use with N-series Azure VMs.
Torch-CPU
This Torch-CPU recipe contains information on how to containerize Torch for use on Azure Batch compute nodes.
Torch-GPU
This Torch-GPU recipe contains information on how to containerize Torch on GPUs for use with N-series Azure VMs.
Genomics and Bioinformatics
BLAST-CPU
This BLAST-CPU recipes contains information on how to containerize a BLAST pipeline with parallelization using Azure Batch compute nodes.
RNASeq-CPU
This RNASeq-CPU recipes contains information on how to containerize a proof-of-concept RNA-Seq pipeline using Azure Batch compute nodes.
Molecular Dynamics (MD)
NAMD-GPU
This NAMD-GPU recipe contains information on how to containerize NAMD on GPUs for use with N-Series Azure VMs.
NAMD-Infiniband-IntelMPI
This NAMD-Infiniband-IntelMPI recipe contains information on how to containerize distributed NAMD across Infiniband/RDMA Azure VMs with Intel MPI.
NAMD-TCP
This NAMD-TCP recipe contains information on how to containerize distributed NAMD across multiple Azure Batch compute nodes using TCP.
RemoteFS Provisioning
RemoteFS-GlusterFS
This RemoteFS-GlusterFS recipe contains information on how to provision a sample multi-VM GlusterFS storage cluster.
RemoteFS-GlusterFS+BatchPool
This RemoteFS-GlusterFS+BatchPool recipe contains information on how to provision a Batch pool and automatically link it against a provisioned GlusterFS storage cluster.
RemoteFS-NFS
This RemoteFS-NFS recipe contains information on how to provision a sample single VM NFS server.
Slurm on Batch
Slurm+NFS
This recipe contains information on how to orchestrate a Slurm cluster with an NFS shared file system.
Video Processing
FFmpeg-GPU
This recipe contains information on how to containerize FFmpeg on GPUs for use with the N-series Azure VMs.
Windows
DiskSpd-Windows
This recipe contains information on how to run a containerized version of the DiskSpd tool with Windows Server Containers on Azure VMs.
DotNet-Windows
This recipe contains information on how to run a sample DotNet application with Windows Server Containers on Azure VMs.