arXiv:2605.24006cs.DCcs.LG2026-05中稿 · the 25th IEEE Inte…

提出表格化调度抽象,揭示通信对大模型流水并行调度效果的决定性影响。

A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training

论文配图:A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training
图 1 · 摘自论文原文
  • 构建表格化调度抽象,融合公式分析与通信感知仿真。
  • 1F1B比GPipe更省激活内存,但运行时间相当;Chimera仅在低微批时有效。
  • 强调调度优劣必须结合具体通信环境判断,适合系统优化研究者。

流水并行是分布式训练大语言模型的关键技术,可降低单设备的参数和激活内存占用。然而,比较不同流水调度方案困难:解析模型仅暴露如气泡率等结构指标,而端到端硬件实验成本高且依赖具体系统。本文提出一种表格化调度抽象与统一多抽象方法,连接公式推理、理想调度表与通信感知执行仿真。基于该框架,我们在多种建模系统配置下对比了GPipe、1F1B、Chimera和Hanayo在受限场景下的表现。结果表明,调度排名并非抽象无关:通信可能抵消仅凭气泡分析得出的结构优势。在当前假设下,GPipe与1F1B运行时间相当,但1F1B激活内存峰值更低;Chimera主要在微批较小及通信有利条件下占优;Hanayo在其设计的受限操作点有效,但仍易受网络瓶颈影响。我们进一步研究了非对称式Chimera布局,虽未降低全局峰值内存,但在浅层流水中带来有限运行时间收益。总体而言,流水调度质量需在特定建模执行环境中评估。

原文摘要 · Abstract (English)

Pipeline parallelism is a key technique for distributed training of large language models because it reduces per-device parameter and activation memory. However, comparing pipeline schedules is difficult: analytical models expose structural quantities such as bubble ratios, while end-to-end hardware experiments are costly and system-specific. In this work, we introduce a tabular schedule abstraction and a unified multi-abstraction methodology that connects formula-based reasoning, idealized schedule tables, and communication-aware execution simulation. Using this framework, we compare GPipe, 1F1B, Chimera, and Hanayo in its restricted regime across multiple modeled system configurations. Our results show that schedule rankings are not abstraction-invariant: communication can negate structural advantages suggested by bubble analysis alone. Under the assumptions considered here, GPipe and 1F1B are runtime-equivalent, but 1F1B achieves a lower activation-memory peak. Chimera is advantageous mainly at low microbatch counts and in communication-favorable regimes, while Hanayo is effective in its intended restricted operating point but remains sensitive to network bottlenecks. We further study an asymmetric Chimera-style placement, which does not reduce the global peak memory requirement but reveals limited runtime gains in shallow pipelines. Overall, pipeline schedule quality is meaningful only in the context of the modeled execution environment.

大模型训练流水并行通信优化调度抽象

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。