diff --git a/docs/_data/navigation.yml b/docs/_data/navigation.yml index 5898432fe..afdc7e218 100755 --- a/docs/_data/navigation.yml +++ b/docs/_data/navigation.yml @@ -39,6 +39,8 @@ lnav: url: /docs/config-json/#communication-options - title: "FP16" url: /docs/config-json/#fp16-training-options + - title: "BFOAT16" + url: /docs/config-json/#bfloat16-training-options - title: "Gradient Clipping" url: /docs/config-json/#gradient-clipping - title: "ZeRO optimizations" diff --git a/docs/_pages/config-json.md b/docs/_pages/config-json.md index 84a7cd1fc..5f81f2086 100755 --- a/docs/_pages/config-json.md +++ b/docs/_pages/config-json.md @@ -233,7 +233,7 @@ Example of **scheduler** | ----------------------------------------------------------------------------------------------------- | ------- | | **min_loss_scale** is a **fp16** parameter representing the minimum dynamic loss scale value. | `1000` | -### BFLOAT16 options +### BFLOAT16 training options **Note:** this mode cannot be combined with the `amp` mode described below. {: .notice--warning} @@ -241,10 +241,7 @@ Example of **scheduler** **Note:** this mode cannot be combined with the `fp16` mode described above. {: .notice--warning} -**Note:** this mode is only compatible with ZeRO stages 1 and 2. -{: .notice--warning} - -**bfloat16**: [dictionary] +**bf16**: [dictionary] | Description | Default | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | @@ -256,11 +253,11 @@ Example of **scheduler** } ``` -**bfloat16:enabled**: [boolean] +**bf16:enabled**: [boolean] -| Description | Default | -| ------------------------------------------------------------------------------------------- | ------- | -| **enabled** is a **bfloat16** parameter indicating whether or not BFLOAT16 training enabled. | `false` | +| Description | Default | +|--------------------------------------------------------------------| ------- | +| **enabled** indicates whether BFLOAT16 training is enabled. | `false` | ### Automatic mixed precision (AMP) training options