让不同品牌的显卡协同训练大模型，性能不降反而提升
HetCCL: Accelerating LLM Training with Heterogeneous GPUs
- 统一不同厂商显卡的通信接口，支持跨品牌高效协同
- 在异构集群中实现与同质环境相当的训练速度
- 无需修改代码或驱动，适合已有多品牌GPU的团队
大语言模型的快速发展推动组织扩展多厂商GPU集群，但现有深度学习框架缺乏对异构显卡间集体通信的支持，导致效率低下、成本上升。我们提出HetCCL，一个统一各厂商通信后端的集体通信库，可在不修改驱动的前提下实现基于RDMA的跨厂商显卡通信。HetCCL引入两项新机制，在充分利用NVIDIA NCCL和AMD RCCL优化库的同时，实现跨厂商通信。在多厂商GPU集群上的评估显示，HetCCL在同质环境下性能与NCCL和RCCL相当，而在异构环境中具有唯一可扩展性，使使用NVIDIA和AMD显卡的深度学习应用无需修改即可实现高效训练。
原文摘要 · Abstract (English)
The rapid growth of large language models is driving organizations to expand their GPU clusters, often with GPUs from multiple vendors. However, current deep learning frameworks lack support for collective communication across heterogeneous GPUs, leading to inefficiency and higher costs. We present HetCCL, a collective communication library that unifies vendor-specific backends and enables RDMA-based communication across GPUs without requiring driver modifications. HetCCL introduces two novel mechanisms that enable cross-vendor communication while leveraging optimized vendor libraries, NVIDIA NCCL and AMD RCCL. Evaluations on a multi-vendor GPU cluster show that HetCCL matches NCCL and RCCL performance in homogeneous setups while uniquely scaling in heterogeneous environments, enabling practical, high-performance training with both NVIDIA and AMD GPUs without changes to existing deep learning applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。