提出新算法，让故障时的分布式训练速度接近无故障状态。
Don't Let a Few Network Failures Slow the Entire AllReduce
- 设计四阶段流水线算法，动态适应不同带宽的故障节点。
- 在50%带宽损失下，比现有方法快2-6%，仅比理想情况慢少许。
- 适合大规模训练中需高容错性的场景，如科研与工业级集群。
网络故障是大规模GPU集群中最常见的硬件故障之一，也是训练任务中断的主要原因。现代集体通信库（如NCCL）通过将流量重路由至同一服务器的存活网卡来缓解故障，以牺牲跨节点带宽为代价维持训练连续性。然而，性能较差的服务器仍处于标准环形算法的关键路径上，拖累整体进度。本文首次给出在异构带宽条件下AllReduce完成时间的信息论下界，证明当慢节点保留至少一半原始带宽时，相对于无故障最优解的不可避延迟仅为O(1/p)（p为GPU数量）。基于此，我们设计了OptCC——一种四阶段流水线式AllReduce算法，逼近该理论下界。在SimAI上的实验表明，当网络故障导致最高50%带宽损失时，OptCC的性能仅比NCCL在无故障时低2-6%，而当前最先进的方案则可能高达57%的开销。
原文摘要 · Abstract (English)
Network failures are among the most frequent hardware faults in large-scale GPU clusters and a leading cause of training-job interruptions. Modern collective communication libraries such as NCCL mitigate network failures by rerouting traffic through surviving NICs on the same server, trading reduced inter-node bandwidth for uninterrupted training. However, the degraded server remains on the critical path of the standard ring algorithm, slowing the entire collective. We present the first information-theoretic lower bound on AllReduce completion time under asymmetric network bandwidth and show that when the straggler retains at least half of its original bandwidth, the unavoidable overhead relative to the fault-free optimum is only O(1/p) for p GPUs. We then design OptCC, a four-stage pipelined AllReduce algorithm that approaches this lower bound. Experiments on SimAI confirm that OptCC closes the gap left by existing fault-tolerant schemes: under practical network failures with up to 50% bandwidth loss, OptCC completes AllReduce within 2-6% of NCCL's fault-free ring performance, whereas the state-of-the-art incurs up to 57% overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。