新通信库PCCL让千卡规模深度学习训练更快更稳
The Big Send-off: Scalable and Performant Collectives for Deep Learning
- 用分层设计+自适应算法选最优通信方案
- 在前沿超算上提速最高达168倍，比RCCL快10倍以上
- 适合大规模分布式训练，尤其对大模型训练者有用
随着数据中心和超级计算机中分布式人工智能任务增多，集体通信的重要性日益凸显。然而现有通信库如NCCL、RCCL和Cray-MPICH在现代GPU超算上存在性能与可扩展性瓶颈。为此，我们提出面向分布式深度学习的高性能通信库PCCL，针对all-gather、reduce-scatter、all-reduce等关键操作提供高度优化实现。PCCL采用分层架构，结合学习驱动的自适应算法选择最佳执行路径，可在数千张GPU上高效扩展。在前沿超算（Frontier）上，对2048个GCD的测试显示：reduce-scatter最高提速168倍，all-gather达33倍，all-reduce达10倍；在Perlmutter系统上相比NCCL提升最高达5.7倍。这些性能提升直接转化为实际训练加速：DeepSpeed ZeRO-3训练最快提速4.9倍，DDP训练最快提速2.4倍。
原文摘要 · Abstract (English)
Collective communication is becoming increasingly important in data center and supercomputer workloads with an increase in distributed AI related jobs. However, existing libraries that provide collective support such as NCCL, RCCL, and Cray-MPICH exhibit several performance and scalability limitations on modern GPU supercomputers. To address these challenges, we introduce the Performant Collective Communication Library (PCCL), specifically targeted for distributed deep learning (DL) workloads. PCCL provides highly optimized implementations of key collectives used in distributed DL: all-gather, reduce-scatter, and all-reduce. PCCL uses a hierarchical design with learning-based adaptive selection of the best performing algorithms to scale efficiently to thousands of GPUs. It achieves substantial performance speedups over RCCL on 2048 GCDs of Frontier -- up to 168x for reduce-scatter, 33x for all-gather and 10x for all-reduce. More modest but still significant gains up to 5.7x over NCCL are observed on Perlmutter. These gains translate directly to performance improvement of production DL workloads: up to 4.9x speedup over RCCL in DeepSpeed ZeRO-3 training, and up to 2.4x speedup in DDP training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。