Set the default value of op_builder/xxx.py/is_compatible()/verbose to
False for quite warning.
Add verbose judgement before
op_builder/xxx.py/is_compatible()/self.warning(...).
Otherwise the verbose arg will not work.
---------
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
**Auto-generated PR to update version.txt after a DeepSpeed release**
Released version - 0.15.1
Author - @loadams
Co-authored-by: loadams <loadams@users.noreply.github.com>
This PR fixes an issue addressed in #5921.
With this change, we only apply the patch for parameter partitioning to
classes that have `__init__` so that we can avoid applying the patch
multiple times.
The class that does not have `__init__` now uses its superclass's one.
So this PR also applies the patch to the root class,
`torch.nn.modules.module.Module`.
Thanks @VeryLazyBoy for the report and initial solution.
---------
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
move the logic that prints a warning when triton cache dir is on NFS to
act on the actual calculated cache_dir rather than on the default.
this means that:
- when the default directory (in the user's home directory) is on NFS
but `TRITON_CACHE_DIR` is set to a non-NFS directory, no warning will be
printed whereas prior to this change a spurious and confusing warning
was printed
- when the user's home directory is not on NFS but `TRITON_CACHE_DIR` is
set to an NFS directory, a warning will be printed whereas prior to this
change no warning would be printed
fixes#6486
- fix step function to cast to FP32 before step in case of different
gradient accumulation data type
- remove redundatn function initialize_optimizer_states()
- Adds a nightly workflow that tests to confirm we can build DeepSpeed
without torch as a dependency, as this often only surfaces when doing a
release.
This PR is to avoid the below error during DeepSpeed build on ROCm. The
error is because of the incompatibility of GDSBuilder extension on ROCm.
```
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-req-build-lv1v39xc/setup.py", line 180, in <module>
op_compatible = builder.is_compatible()
File "/tmp/pip-req-build-lv1v39xc/op_builder/gds.py", line 47, in is_compatible
CUDA_LIB64 = os.path.join(CUDA_HOME, "lib64")
File "/opt/conda/envs/py_3.9/lib/python3.9/posixpath.py", line 76, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Total number of unsupported CUDA function calls: 0
Total number of replaced kernel launches: 1
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output
```
cc: @jithunnair-amd
---------
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
previous condition check is not right, it would cause this condition
always be True.
---------
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
The `numThreads` config option determines how many threads are used to
read from the file. In the CPU case these threads are created via AIO,
in the GDS case they are handled by the GDS library via the cufile.json.
If we were to also create AIO threads it would have a multiplicative
effect. Example 8 AIO threads * 8 GDS threads would be 64 threads
reading from the file when the user really only intended for 8 threads.
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Tested with triton==3.0.x and the kernel tests pass so adding as an
allowed version.
Triton 2.3.x is not well supported on arm64. Triton 3.0.0 is supported
on arm64 and it appears the fp8 kernel works fine with triton==3.0.0 so
this simplifies usage on arm hosts (GH200).
CIFAR10_DATASET_PATH -> Path where the dataset is stored
STORE_CIFAR10 -> Store the dataset 1/0
CIFAR10_OFFLINE -> To use offline dataset 1/0
MISC:
Added getDeviceId to get device if by name in case of accelerator
---------
Co-authored-by: Shaik Raza Sikander <srsikander@habana.ai>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
Co-authored-by: Logan Adams <loadams@microsoft.com>
Check if the dtype is supported by the accelarator if not then skip
---------
Co-authored-by: Shaik Raza Sikander <srsikander@habana.ai>
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
This PR adds the setup instructions for Huawei Ascend NPU. Please refer
to the remainder of the guide for instructions on other devices.
---------
Co-authored-by: sjh <sjh1270@163.com>
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
Co-authored-by: Logan Adams <loadams@microsoft.com>
Deprecate redundant sequence_data_parallel_group argument. Users/client
code will control across which process group Z3 parameters will be
partitioned from one of [None, data_parallel_group,
sequence_data_parallel].
---------
Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
`num_bytes_per_thread` was a smaller type than `file_num_bytes`, this
caused issues when dividing by `num_threads`.
Co-authored-by: jomayeri <deepspeed@H100-VM2.shlnn55tgwve1eacvp21ie45dg.jx.internal.cloudapp.net>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
**Auto-generated PR to update version.txt after a DeepSpeed release**
Released version - 0.15.0
Author - @loadams
Co-authored-by: loadams <loadams@users.noreply.github.com>
Pydantic v2 has been out for some time now. We have been relying on
using the v1 API available in v2 until now. This is a refresh of #3902
to bring proper v2 support to DeepSpeed.
Corresponding DeepSpeed-MII PR
[here](https://github.com/microsoft/DeepSpeed-MII/pull/423).
@loadams
---------
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
Co-authored-by: Logan Adams <loadams@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Abhishek Kulkarni <11399+adk9@users.noreply.github.com>
Co-authored-by: Abhishek Kulkarni <abkulkarni@microsoft.com>
Co-authored-by: Lev Kurilenko <113481193+lekurile@users.noreply.github.com>
with intel_extension_for_pytorch new release, we changed deepspeed
kernel API. Need to align with upstream DeepSpeed xpu op builder.
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
based on PR #5403 (Qwen1.5-MOE) and #5219 (Qwen1.5), support Qwen2
series model.
including: 0.5B, 1.5B, 7B, 57B-A14B, and 72B models.
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
The operation `.to('cpu') `is not necessary for exp_counts, and it will
cause device to host synchronization which damage performance.
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
This PR enhances capabilities of [DeepSpeed long sequence (context)
parallelism (aka DS
Ulysses)](https://dl.acm.org/doi/10.1145/3662158.3662806) with support
for HuggingFace (and by extension other frameworks) models. With HF
integration, users can use sequence parallelism for model
pre/mid/post-training, finetuning etc. Usage requires both _torch
>=2.2.2 and flash-attention_. ZeRO-1 and 2 are supported, ZeRO-3 and
SPDA support in progress. Corresponding PR in HF is
[PR32305](https://github.com/huggingface/transformers/pull/32305).
---------
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
Allowing hf args to be passed through class to AutoConfig.pretrained.
Co-authored-by: Ubuntu <deepspeed@H100-VM2.shlnn55tgwve1eacvp21ie45dg.jx.internal.cloudapp.net>
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
DeepSpeed on Windows blog
---------
Co-authored-by: Logan Adams <loadams@microsoft.com>
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
nv-nightly was failing due to updates in transformers, we will need to
introduce a real fix for these, but this at least gets the test passing
and we need to update transformers support for MII too.
Gives the ability to add and remove the forward hooks in ZeRO 3 by using
a context manager. These code changes were taken from a Huggingface
[PR](https://github.com/huggingface/trl/pull/1617) and integrated for
direct support in DeepSpeed.
This is useful in the inference case and the speedup can be observed
[here](https://github.com/huggingface/trl/pull/1483).
---------
Co-authored-by: root <root@deepspeed-c000004.2d1icxc5dsxehnpuwt3ifc34ph.gvxx.internal.cloudapp.net>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Heyang Qin <heyangqin@microsoft.com>
Co-authored-by: Logan Adams <loadams@microsoft.com>
This PR enables dynamic shapes for inputs to pipeline parallel (PP)
engine.
Currently PP engine checks tensor shapes and allocates communication
buffer at the first forward/backward passes. This causes a tensor shape
mismatch error when input tensor shapes changed.
This PR adds an option to check tensor shapes at every iteration and
allocate buffer based on the shapes. As shown below, you can enable this
feature by passing `dynamic_shape=True` to `PipelineModule`.
Note that this might have a performance impact and the option is set to
False as default.
```python
model = PipelineModule(
...
dynamic_shape=True
)
```
This will increase the overhead of buffer allocation and communication
for tensor metadata. To mitigate the overhead, this PR also includes
these improvements:
- Consolidate multiple communication calls to send/recv tensor shapes
9f96ad4049
- Reuse (extend) communication buffer instead of creating a new one
b3c07504be
---------
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Must use the same device as grad_partitions_flat_buffer
---------
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Notice some users need to use topk > 2 to train MoE models. For example:
https://huggingface.co/Qwen/Qwen2-57B-A14B/blob/main/config.json, this
PR adds support for topk (k > 2) gates.
- add topk (k>2) support
- add drop token policy based on position and probabilities.
- unit tests
---------
Co-authored-by: Kurt Chen <kurt.chen@intel.com>
Co-authored-by: Jin, Youzhi <youzhi.jin@intel.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
**Auto-generated PR to update version.txt after a DeepSpeed release**
Released version - 0.14.5
Author - @loadams
Co-authored-by: loadams <loadams@users.noreply.github.com>
Nadav Elyahu
Jinxing Pan
Logan Adams
Alex Morehead
Olatunji Ruwase
Jiancheng Liu
Masahiro Tanaka
Joshua C. Randall
Raza Sikander
Ramya Ramineni
Yizhou Wang
Joe Mayer
Siddartha Naidu
Roger Feng
Dogacan Colak
Logan Adams
jiahao su
Sam Ade Jacobs
Michael Wyatt
Guanhua Wang
Perry Zou
ranzhejiang
Liran Bachar
inkcherry
Rohan Potdar