Piper让用户轻松组合多种并行策略,自动优化训练效率。
Piper: A Programmable Distributed Training System

- 用注解和指令声明并行策略,系统自动编译执行计划。
- 在DeepSeek-V3的双流水线策略下实现计算通信联合调度优化。
- 适合需要灵活部署新并行方案的研究者与工程师使用。
大规模模型训练依赖多种并行策略(如数据并行、流水线并行、专家并行)与内存优化技术(如ZeRO)的组合。现有系统通常需人工设计高层并行策略并实现底层执行逻辑,难以适应新策略;而通用框架虽灵活但受限于固定并行模式,难以集成前沿技术。本文提出Piper,一个用户可控的分布式训练系统,将策略与运行时实现解耦。用户通过少量模型注解和调度指令声明完整训练策略,系统基于统一的中间表示(IR)——全局训练有向无环图(DAG),对计算与通信进行转换。该IR用于生成每设备的执行计划,并由与策略无关的分布式运行时执行。实验表明,Piper在主流策略(如ZeRO)上保持性能一致,同时在复合并行策略(如DeepSeek-V3的DualPipe)中实现计算与通信的联合调度,带来额外的性能与内存效率提升。
原文摘要 · Abstract (English)
Large-scale model training increasingly relies on composing multiple parallelism strategies, such as data, pipeline, and expert parallelism, together with memory-saving optimizations like ZeRO. Deployed systems for foundation model pretraining often rely on human experts to manually design a high-level parallelism strategy then implement the corresponding low-level execution strategy, making it difficult to adapt the system to new strategies. Meanwhile, many general-purpose frameworks are more flexible but their implementations are still tied to a fixed set of common parallelism strategies, making it challenging to integrate state-of-the-art strategies. We present Piper, a user-controllable distributed training system that decouples the strategy from the runtime implementation. Piper allows users to declare a comprehensive distributed training strategy with a small set of model annotations and scheduling directives. Each directive applies a transformation on Piper's intermediate representation (IR), a unified global training DAG that represents all computation and communication. Using this IR, Piper compiles per-device execution plans and executes them with a distributed runtime agnostic to the strategy. We show that the combined system maintains performance parity on commonly available strategies such as ZeRO, while also enabling additional performance and memory efficiency gains through joint scheduling of compute and communication in composed parallelism strategies such as DeepSeek-V3's DualPipe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。