Piper通过资源建模与流水线并行，显著提升大规模MoE模型训练效率。
Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism
- 基于资源建模动态优化MoE训练策略，解决异构平台瓶颈。
- 实现2-3.5倍更高计算利用率（MFU），通信带宽提升1.2-9倍。
- 适合超大规模MoE模型在高性能计算平台上的高效训练。
前沿模型越来越多采用混合专家（MoE）架构，在降低资源成本的同时实现大模型性能。然而，在高性能计算（HPC）平台上训练MoE模型面临巨大内存开销、跨异构网络的频繁大规模通信以及严重的负载不均衡问题。为此，我们建立数学模型，量化不同并行策略下MoE配置的内存、计算与通信需求，并通过微基准测试、代码插桩和硬件分析验证。分析发现主要瓶颈包括：专家并行带来的大规模全对全通信延迟、计算与通信重叠不足、由负载不均导致的瘦小GEMM计算效率低下，以及缺乏平台感知的混合并行策略。为应对这些问题，我们提出Piper框架，利用资源建模识别目标HPC平台上的高效训练策略，并引入流水线并行与优化调度。Piper相比X-MoE等先进框架实现2-3.5倍更高的平均浮点利用率（MFU），新提出的全对全算法带来1.2-9倍的带宽提升。
原文摘要 · Abstract (English)
Frontier models increasingly adopt Mixture-of-Experts (MoE) architectures to achieve large-model performance at reduced cost. However, training MoE models on HPC platforms is hindered by large memory footprints, frequent large-scale communication across heterogeneous networks, and severe workload imbalance. To characterize these challenges, we develop a mathematical model that quantifies memory, compute, and communication requirements for MoE configurations under various parallelization schemes, verified through micro-benchmarking, code instrumentation, and hardware profiling. Our analysis identifies performance bottlenecks: all-to-all latency at scale from expert parallelism, insufficient compute-communication overlap, low GPU utilization from imbalanced skinny GEMMs, and the absence of platform-aware hybrid parallelization strategies. To address these, we introduce Piper, a framework that leverages resource modeling to identify efficient training strategies for MoE models on target HPC platforms, applying pipeline parallelism with optimized schedules. Piper achieves 2-3.5X higher MFU than state-of-the-art frameworks such as X-MoE, and a novel all-to-all algorithm delivers 1.2-9X bandwidth over vendor implementation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。