提出异步多方向流水线并行，提升大规模模型训练速度且不损失收敛性。
AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training
- 限制流水线首阶段最多处理两个小批量，控制前后向参数差异。
- 通过并发多流水线和梯度累积，实现训练加速且误差仅限于单步优化。
- 适合需要高吞吐量的大规模语言模型训练场景。
流水线并行对大规模模型训练至关重要，但现有异步方法常因前向与反向传递间参数不一致导致收敛性下降。本文提出异步多方向流水线并行（AMDP），在保持高利用率的同时缓解此问题。AMDP 限定每个流水线首阶段最多处理两个小批量后才进行反向传播，从而限制前后向之间的参数更新次数。为减少由此产生的流水线空洞，AMDP 启动多个并发流水线，并根据流水线深度动态调整数量。此外，AMDP 在小批量间累积梯度，并在单次更新中应用，确保仅有有限数量的小批量存在参数不一致，且该范围被限制在单个优化步骤内。在 GPT 与 BERT 风格模型上的实验表明，AMDP 显著加速训练过程，同时维持良好收敛性。
原文摘要 · Abstract (English)
Pipeline parallelism is essential for large-scale model training, but existing asynchronous approaches often degrade convergence due to parameter mismatch between forward and backward passes. We propose Asynchronous Multi-Directional Pipeline parallelism (AMDP) to mitigate this issue while sustaining high utilization. AMDP limits the first stage of each pipeline to process at most two minibatches before backpropagation, bounding the number of parameter updates between forward and backward passes. To alleviate the resulting pipeline bubbles, AMDP launches multiple concurrent pipelines and adapts their number according to pipeline depth. In addition, AMDP accumulates gradients across minibatches and applies them in a single update, ensuring that only a bounded number of minibatches experience parameter mismatch, limited to within one optimization step. Experiments on GPT- and BERT-style models demonstrate that AMDP significantly accelerates training while preserving convergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。