动态调度提升大模型训练效率,减少等待空洞。
A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability

- 根据任务就绪状态动态调度,不强制执行预设顺序。
- 语言模型最高提速1.77倍,多模态最高达2.77倍。
- 适合大规模分布式训练,尤其在异构算力环境下。
流水线并行是扩展大模型训练的关键技术,但现代工作负载在计算和通信上存在运行时波动。现有流水线系统通常采用静态、采样或自适应生成的调度作为预设执行顺序。当实际任务就绪情况与预设顺序不符时,各阶段可能等待尚未就绪的任务,即使其他可执行任务已就绪,导致阶段错位、空闲气泡和利用率下降。我们提出运行时就绪优先流水线(RRFP),一种面向运行时变异的流水线并行运行时。RRFP 改变调度的使用方式:不再将调度视为必须遵循的序列,而是作为当前就绪任务的非强制排序提示。为支持此机制,RRFP结合消息驱动的异步通信、轻量级张量并行协调以保证集合操作一致性,以及就绪-就绪仲裁实现低开销分派。我们在基于 Megatron 的训练框架中实现了 RRFP,评估了语言模型与多模态任务在最多 128 张 GPU 上的表现。在所有设置下,RRFP 均优于固定顺序流水线基线。使用 BFW 提示时,语言模型最高提速 1.77×,多模态最高达 2.77×。跨框架对比中,使用默认 BF 提示的 RRFP 比现有更快外部系统最高快 1.84×,同时保持训练正确性。
原文摘要 · Abstract (English)
Pipeline parallelism is a key technique for scaling large-model training, but modern workloads exhibit runtime variability in computation and communication. Existing pipeline systems typically consume static, profiled, or adaptively generated schedules as pre-committed execution orders. When realized task readiness diverges from the pre-committed order, stages may wait for not-yet-ready work even though other executable work is available, creating stage misalignment, idle bubbles, and reduced utilization. We present Runtime-Readiness-First Pipeline (RRFP), a readiness-driven runtime for pipeline-parallel training. RRFP changes how schedules are consumed at runtime: instead of treating a schedule as a sequence that stages must wait to follow, it treats the schedule as a non-binding hint order for ranking currently ready work. To support this model, RRFP combines message-driven asynchronous communication, lightweight tensor-parallel coordination for collective consistency, and ready-set arbitration for low-overhead dispatch. We implement RRFP in a Megatron-based training framework and evaluate it on language-only and multimodal workloads at up to 128 GPUs. RRFP improves over fixed-order pipeline baselines across all settings. Using the BFW hint, RRFP achieves up to 1.77$\times$ speedup on language-only workloads and up to 2.77$\times$ on multimodal workloads. In cross-framework comparisons, RRFP with the default BF hint outperforms the faster available external system by up to 1.84$\times$ while preserving training correctness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。