CycleCloud project to enable configuration, orchestration, and management of BeeGFS file systems in Azure CycleCloud HPC clusters.
Перейти к файлу
anhoward 1fde1ed00d
Merge pull request #16 from Azure/users/GitHubPolicyService/7c275ed7-2e2f-4ee7-882e-32df3932c49c
Adding Microsoft SECURITY.MD
2022-09-28 15:33:05 -04:00
examples add client OFED compatibilty 2021-01-27 10:23:27 -08:00
images * update to 7.1.2 (point to latest release) 2019-03-07 15:01:58 -08:00
specs/default modified azvolumes setup scripts to use parted instead of fdisk to allow 2021-01-26 10:11:03 -05:00
templates add client OFED compatibilty 2021-01-27 10:23:27 -08:00
.gitignore Ignore build directory in git 2019-03-07 10:21:39 -08:00
LICENSE Initial commit 2018-07-28 12:56:09 -07:00
README.md make local disk the default 2019-07-22 15:31:05 -07:00
SECURITY.md Microsoft mandatory file 2022-07-28 16:53:30 +00:00
icon.png Remove icon url from template and add icon to repo 2018-08-11 01:49:59 +08:00
project.ini add client OFED compatibilty 2021-01-27 10:23:27 -08:00

README.md

Using CycleCloud for setting up a BeeGFS Cluster on Azure

A CycleCloud Project for starting a BeeGFS cluster in Azure.

Performance, Capacity and Cost

This project includes two scenarios; one configuration based on low-latency local NVMe SSDs and one based on Azure Premium Disks.

Scenario Local Disk Persistent Disk
VM Size Standard_L8s_v2 Standard_D16s_v3
Disk Local NVMe SSD Premium Disk (2 x P30)
Capacity 1.9 TB 2.0 TB
Node Throughput 400 MB/s 384 MB/s
Node IOPS 40000 10000
Node $/month $455 $830
Node Data Durability 3 9's 11 9's

These configurations have been designed and tested to have maximal performance-cost. The Local Disk scenario will show better performance, particularly latency and IOPS, but durability of data will be relatively lower. The performance (and capacity) of a cluster can be estimated as the performance of a single node multiplied by the number of storage nodes in the cluster. This has been tested up to 32 nodes.

If cost-per-storage is more important that cost-per-performance then you can increase the size or quantity of additional premium disks attached. Increasing the attached disks will increase the capacity, but not the performance.

Using the persistent disks, the data on the cluster will be recoverable even in the event of unplanned node termination or VM scheduled maintenance. This is not so when using local disks as the disk is only stored locally and will be lost when the VM is deallocated.

Metadata

BeeGFS is not fully supporting extended file attributes. Applications that make heavy use of setxattr and getxattr can expose instabilities in the beegfs-meta daemon. Basic metadata attributes are efficiently used but one should avoid using BeeGFS for applications using extended file attributes.

Cluster Life-Cycle for Local Disk (default)

For storage cluster based on local SSDs storage is not preserved across VM deallocations and restarts.

  • Create Cluster - creates storage VMs and disks
  • Add Node - add additional node will increase size & resources of cluster
  • Shutdown/Delete Node - delete VM and disks, data on disks will be destroyed.
  • Terminate Cluster - delete all VMs and disks, all data destroyed.

Cluster Life-Cycle for Premium Disk

It is possible to delete data and disks managed by CycleCloud in the CycleCloud UI. This can result in data loss. Here are the actions available in the CycleCloud management.

  • Create Cluster - creates storage VMs and disks
  • Add Node - add additional node will increase size & resources of cluster
  • Shutdown/Deallocate Node - will suspend node but preserve disks.
  • Start Deallocated Node - restore data and resources of deallocated node.
  • Shutdown/Delete Node - delete VM and disks, data on disks will be destroyed.
  • Terminate Cluster - delete all VMs and disks, all data destroyed.

It is possible to create a BeeGFS cluster, populate the data then when the workload is finished, deallocate the VMs so that the cluster can be restarted. This is helpful in controlling costs, because charges for the VMs will be suspended while deallocated. Keep in mind that disks will still accrue charges while the VMs are deallocated.

CC VM Deallocate

Monitoring

This cluster includes a recipe for I/O monitoring in Grafana. By adding a monitor node to the cluster a VM will start and host a monitoring UI. This service can be accessed by HTTP on port 3000 of the monitor host (username: admin, default_pw: admin).

BeeGFS Monitoring

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.