通过细粒度切片与重叠计算，彻底消除大模型训练通信开销。
Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping
- 将单批次数据依赖拆解为独立小块，实现计算与通信重叠
- 在Nvidia DGX-H100上相比Megatron-LM最高提速1.3倍
- 适用于大规模分布式训练，适合追求极致效率的团队
随着生成式AI的普及，大型语言模型（LLMs）训练常需数百甚至上千块GPU进行并行加速，通信开销在大规模训练中愈发显著。为消除分布式LLM训练中的通信开销，本文提出Domino，一种通用方案，可将通信隐藏于计算之后。通过将单批次训练的数据依赖分解为更小的独立单元，Domino对这些独立单元进行流水线处理，并提供细粒度的通信与计算重叠策略。大量实验表明，相较于Megatron-LM，Domino在Nvidia DGX-H100 GPU上实现高达1.3倍的训练加速。
原文摘要 · Abstract (English)
Given the popularity of generative AI, Large Language Models (LLMs) often consume hundreds or thousands of GPUs for parallelizing and accelerating the training process. Communication overhead becomes more pronounced when training LLMs at scale. To eliminate communication overhead in distributed LLM training, we propose Domino, which provides a generic scheme to hide communication behind computation. By breaking data dependency of a single batch training into smaller independent pieces, Domino pipelines these independent pieces training and provides generic strategy of fine-grained communication and computation overlapping. Extensive results show that, comparing with Megatron-LM, Domino achieves up to 1.3x speedup for LLM training on Nvidia DGX-H100 GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。