arXiv:2510.05186cs.DCcs.AI2025-10被引 1

优化大模型训练的流水线调度,减少内存占用和等待时间。

OptPipe: Memory- and Scheduling-Optimized Pipeline Parallelism for LLM Training

  • 将调度问题建模为联合优化问题,兼顾内存、计算与延迟。
  • 在相同内存限制下,流水线空闲时间减少50%。
  • 适合需要高效训练大模型的团队或资源受限场景。

流水线并行(PP)已成为跨多设备扩展大语言模型(LLM)训练的标准技术。尽管通过激活值卸载减少了内存消耗,现有方法仍主要依赖启发式且粗粒度策略,常忽视内存、计算与调度延迟之间的细粒度权衡。本文从原则性优化视角重新审视流水线调度问题。我们发现,现有策略或依赖静态规则,或过度卸载激活值,未充分考虑内存约束与调度效率的交互作用。为此,我们提出一个联合考虑内存容量、激活重用与流水线气泡最小化的约束优化模型。求解该模型可获得细粒度调度方案,在严格内存预算下减少流水线气泡。该方法可与现有卸载技术互补:相比以往固定模式以内存换时间,本方法根据模型结构与硬件配置动态优化权衡。实验表明,该方法显著提升吞吐量与内存利用率。尤其在相同设备内存限制下,流水线空闲时间最多减少50%,部分情况下可在有限内存内训练更大模型。

原文摘要 · Abstract (English)

Pipeline parallelism (PP) has become a standard technique for scaling large language model (LLM) training across multiple devices. However, despite recent progress in reducing memory consumption through activation offloading, existing approaches remain largely heuristic and coarse-grained, often overlooking the fine-grained trade-offs between memory, computation, and scheduling latency. In this work, we revisit the pipeline scheduling problem from a principled optimization perspective. We observe that prevailing strategies either rely on static rules or aggressively offload activations without fully leveraging the interaction between memory constraints and scheduling efficiency. To address this, we formulate scheduling as a constrained optimization problem that jointly accounts for memory capacity, activation reuse, and pipeline bubble minimization. Solving this model yields fine-grained schedules that reduce pipeline bubbles while adhering to strict memory budgets. Our approach complements existing offloading techniques: whereas prior approaches trade memory for time in a fixed pattern, we dynamically optimize the tradeoff with respect to model structure and hardware configuration. Experimental results demonstrate that our method consistently improves both throughput and memory utilization. In particular, we reduce idle pipeline time by up to 50% under the same per-device memory limit, and in some cases, enable the training of larger models within limited memory budgets.

大模型训练流水线并行内存优化调度优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。