双向交错流水线提升大模型训练速度，32卡下提速最高28%
BitPipe: Bidirectional Interleaved Pipeline Parallelism for Accelerating Large Models Training
- 混合交错与双向流水线，减少微批次计算时间
- 采用V型调度和急促梯度同步，降低通信开销
- 适合大规模模型训练，尤其对分布式训练效率敏感者
随着模型规模扩大，高效分布式训练需求日益迫切。现有同步流水线并行方法仍面临两大问题：周期性刷新导致的流水线空洞，以及流水线阶段增多引发的额外通信开销。为此，我们提出BitPipe，一种双向交错流水线并行方法，以加速大模型训练。具体地，提出将交错流水线与双向流水线融合的混合方案，缩短每个微批次的计算时间，并使更多设备同时执行。引入V型调度与急促梯度同步机制，减少并重叠设备间通信。在最多32张GPU上进行的实验表明，BitPipe相比当前最优同步方法，使GPT风格和BERT风格模型的训练吞吐率提升1.05倍至1.28倍。代码已开源：https://github.com/wuhouming/BitPipe。
原文摘要 · Abstract (English)
With the increasing scale of models, the need for efficient distributed training has become increasingly urgent. Recently, many synchronous pipeline parallelism approaches have been proposed to improve training throughput. However, these approaches still suffer from two major issues, i.e., pipeline bubbles caused by periodic flushing and extra communication due to the increasing number of pipeline stages. To this end, we propose BitPipe, a bidirectional interleaved pipeline parallelism for accelerating large models training. Specifically, a hybrid scheme of fusing interleaved pipelines with bidirectional pipelines is proposed to reduce the computational time of each single micro-batch and multiply the number of devices executing simultaneously. A V-shaped schedule with eager gradient synchronization is introduced to reduce and overlap the communication between devices. Experiments conducted on up to 32 GPUs show that BitPipe improves the training throughput of GPT-style and BERT-style models by 1.05x-1.28x compared to the state-of-the-art synchronous approaches. The code of our implementation is available at https://github.com/wuhouming/BitPipe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。