首次为管道并行提供理论保障，揭示延迟与性能的深层关系。
Demystifying Pipeline Parallelism: First Theory for PipeDream
- 提出随机化管道方法，实现非凸优化的收敛性证明
- 发现稳态延迟随阶段数平方增长，影响收敛速度
- 对比局部SGD，指明不同任务下并行策略的适用场景
训练现代机器学习模型越来越依赖跨多个加速器的计算分布。数据并行仍是主流，常与张量并行结合使用，但当参数、激活或优化器状态无法容纳于单个设备时，模型并行变得不可避免。本文从PipeDream（PD）的角度研究管道模型并行。第一，我们提出随机化管道（RPD），一种陈旧块-SGD抽象，首次为类似PD的方法提供了清晰的非凸收敛保证。第二，我们进行了缩放诊断：证明稳态PD引入的延迟随阶段数S呈$S^2 - S/2 + O(1)$增长，因此收敛定理中的陈旧读取项按$Θ(γ^2 S^4)$缩放，等价于调优率形式下的$Θ(S^4/K)$。第三，我们与LocalSGD对比，其周期性模型平均以权值陈旧换取同步开销。在模拟时间实验中，对于二次目标和小型语言建模任务，PD表现更优；而随着阶段数增加，逻辑回归任务中LocalSGD则逐渐占优。
原文摘要 · Abstract (English)
Training modern machine learning models increasingly requires computation to be distributed across many accelerators. Data parallelism remains the default choice and is often paired with tensor-parallel sharding, but model parallelism becomes unavoidable once parameters, activations, or optimizer states no longer fit on a single device. This paper studies pipeline model parallelism through the lens of PipeDream (PD) (Harlap et al., 2018). Our first contribution is theoretical: we introduce Randomized PipeDream (RPD), a stale block-SGD abstraction that yields, to our knowledge, the first clean nonconvex convergence guarantee for a PD-style method. Our second contribution is a scaling diagnosis: we prove that the delay induced by steady-state PD grows as $S^2 - S/2 + O(1)$ for $S$ stages, so the stale-read contribution in the convergence theorem scales as $Θ(γ^2 S^4)$, equivalently as $Θ(S^4/K)$ in the tuned-rate form. Our third contribution is a comparison with LocalSGD, whose periodic model averaging trades weight staleness for synchronization bubbles. In our reported simulated-time experiments, the better-performing method depends on the objective: PD performs better on the quadratic objective and on a small language-modeling training-loss task, while for logistic regression LocalSGD becomes superior as the number of stages increases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。