DeepSpeed/deepspeed
Yejing-Lai 85132adc31
enable starcode((kv_head=1)) autotp (#4896)
Hi, This PR is aim to enable starcode(kv_head=1) autotp. Please kindly
review. Thanks~

Co-authored-by: Michael Wyatt <michaelwyatt@microsoft.com>
2024-01-05 20:54:15 +00:00
..
autotuning fix: change ==NONE to is (#3923) 2023-07-11 16:56:43 +00:00
checkpoint Universal Checkpoint for Sequence Parallelism (#4752) 2023-12-15 10:22:39 -08:00
comm Fix exception handling in get_all_ranks_from_group() function (#4862) 2023-12-22 11:48:48 -08:00
compression fix: Remove duplicate word the (#4051) 2023-07-27 09:33:13 -07:00
elasticity Re-enable elastic training for torch 2+ (#4010) 2023-07-21 18:21:35 +00:00
inference Add support of Microsoft Phi-2 model to DeepSpeed-FastGen (#4812) 2024-01-03 13:27:06 -08:00
launcher [NPU] load EXPORT_ENV based on different accelerators to support multi-node training on other devices (#4830) 2023-12-19 03:09:10 +00:00
model_implementations Capture short kernel sequences to graph (#4318) 2023-12-20 20:51:36 +00:00
module_inject enable starcode((kv_head=1)) autotp (#4896) 2024-01-05 20:54:15 +00:00
moe Communication Optimization for Large-Scale Training (#4695) 2023-11-21 14:30:43 -08:00
monitor Introduce pydantic_v1 compatibility module for pydantic>=2.0.0 support (#4407) 2023-10-09 11:59:30 -07:00
nebula fix typo with deepspeed/ (#3547) 2023-06-02 00:47:14 +00:00
ops use all_gather_into_tensor instead of all_gather (#4705) 2023-11-28 17:41:46 +00:00
pipe Update DeepSpeed copyright license to Apache 2.0 (#3111) 2023-03-30 17:14:38 -07:00
profiling Update flops profiler to recurse (#4374) 2023-11-21 18:48:58 +00:00
runtime Support cpu tensors without direct device invocation (#3842) 2024-01-05 13:36:17 +00:00
sequence Communication Optimization for Large-Scale Training (#4695) 2023-11-21 14:30:43 -08:00
utils zero_to_fp32.py: Handle a case where shape doesn't have numel attr (#4842) 2024-01-03 19:45:06 +00:00
__init__.py add type checker ignore to resolve that pylance can't resolved noqa annotation (#4102) 2023-08-08 13:30:23 +00:00
accelerator Abstract accelerator (step 2) (#2560) 2023-01-06 23:40:58 -05:00
constants.py Allow env var for timeout (#4405) 2023-10-10 08:56:10 -07:00
env_report.py [CANN] Support cpu offload optimizer for Ascend NPU (#4568) 2023-11-14 13:37:16 +00:00
git_version_info.py add type checker ignore to resolve that pylance can't resolved noqa annotation (#4102) 2023-08-08 13:30:23 +00:00
pydantic_v1.py Introduce pydantic_v1 compatibility module for pydantic>=2.0.0 support (#4407) 2023-10-09 11:59:30 -07:00