解决大模型训练中网络故障导致的效率损失问题。
Reliable and Resilient Collective Communication Library for LLM Training and Serving
- 利用多网卡硬件实现快速连接迁移和负载均衡。
- 故障下训练损耗低于1%，推理损耗低于3%。
- 适合大规模GPU集群训练与服务场景使用。
现代机器学习训练与推理已扩展至数十到数万张GPU，网络故障可能导致10%至15%的GPU算力浪费。常见网络错误与链路波动常引发超时，导致整个任务终止，训练需回滚检查点，推理则需重新处理请求。本文提出R²CCL，一种容错通信库，通过多网卡硬件实现无损、低开销的故障切换。该库具备快速连接迁移、带宽感知负载重分配及鲁棒集体通信算法，在故障下仍能保持计算进度。我们在两台8卡H100 InfiniBand服务器上测试，并通过大规模模拟器在数百张GPU上评估不同故障模式下的表现。结果表明，R²CCL对网卡故障具有极高鲁棒性，训练开销低于1%，推理开销低于3%。相比基线AdapCC与DejaVu，分别提升12.18倍和47倍。
原文摘要 · Abstract (English)
Modern ML training and inference now span tens to tens of thousands of GPUs, where network faults can waste 10--15\% of GPU hours due to slow recovery. Common network errors and link fluctuations trigger timeouts that often terminate entire jobs, forcing expensive checkpoint rollback during training and request reprocessing during inference. We present R$^2$CCL, a fault-tolerant communication library that provides lossless, low-overhead failover by exploiting multi-NIC hardware. R$^2$CCL performs rapid connection migration, bandwidth-aware load redistribution, and resilient collective algorithms to maintain progress under failures. We evaluate R$^2$CCL on two 8-GPU H100 InfiniBand servers and via large-scale ML simulators modeling hundreds of GPUs with diverse failure patterns. Experiments show that R$^2$CCL is highly robust to NIC failures, incurring less than 1\% training and less than 3\% inference overheads. R$^2$CCL outperforms baselines AdapCC and DejaVu by 12.18$\times$ and 47$\times$, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。