OctoPipe联合优化模型拆分、部署与调度，减少异构大模型训练中的流水线空洞。
OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Scheduling
- 构建图模型模拟器，统一建模异构流水线执行
- 迭代式调优器在组合搜索空间中高效找最优解
- 动态执行引擎支持不规则调度，提升通信重叠度
流水线并行广泛用于训练大语言模型（LLMs）。然而，模型架构的日益异构化加剧了流水线空洞，降低训练效率。现有方法通常仅优化流水线调度的一个环节（如拆分、部署或调度），导致大量空洞残留。尽管联合优化前景广阔，但仍面临三大挑战：（1）复杂性能建模，（2）组合搜索空间庞大，（3）执行顺序不规则。为此，我们提出OctoPipe，一个联合优化拆分、部署与调度的流水线并行系统。首先，构建基于图的流水线模拟器，用于建模异构流水线执行；其次，在模拟器基础上开发迭代式空洞感知调优器，高效探索组合搜索空间；最后，实现统一流水线执行器，动态协调计算与通信，支持不规则执行顺序且无死锁，最大化通信-计算重叠。实验表明，OctoPipe在多种模型和GPU集群规模下，相较当前最优流水线并行方法，吞吐量提升1.15至1.44倍。
原文摘要 · Abstract (English)
Pipeline parallelism is widely used to train large language models (LLMs). However, increasing heterogeneity in model architectures exacerbates pipeline bubbles, thereby reducing training efficiency. Prior approaches typically optimize a single phase of the pipeline schedule (i.e., partitioning, placement, or scheduling), leaving substantial pipeline bubbles. While promising, co-optimization poses three key challenges: (1) complex performance modeling, (2) a combinatorial search space, and (3) irregular execution orders. To address these challenges, we propose OctoPipe, a pipeline parallelism system to jointly optimize partitioning, placement, and scheduling. First, we build a graph-based pipeline simulator to model heterogeneous pipeline execution for co-optimization. Second, on top of the simulator, we develop an iterative bubble-aware tuner to efficiently explore the combinatorial search space. Third, we implement a unified pipeline executor that dynamically orchestrates computation and communication to support irregular execution orders without deadlocks while maximizing communication-computation overlap. Experiments show that OctoPipe achieves 1.15--1.44x throughput improvement over the state-of-the-art pipeline parallelism approaches across various models and GPU cluster scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。