通过利用GPU执行时间差异，加速分布式训练中的梯度同步。
Efficient AllReduce with Stragglers
- 在慢节点等待时并行执行ReduceScatter，提前完成部分计算。
- 8卡服务器上比现有算法快25%，大集群理论速度提升2倍。
- 适合大规模分布式训练，尤其对异步执行敏感的场景
分布式机器学习使用数据并行和张量并行进行训练与推理，二者均依赖AllReduce集合操作来同步梯度或激活值。然而，传统AllReduce受最慢的GPU阻塞，导致整体延迟。为解决此问题，本文提出StragglAR：一种利用GPU执行时间自然差异的并行化AllReduce算法。在慢节点引起的延迟期间，该算法在剩余GPU间执行ReduceScatter，随后采用新型集合操作，在最后一个GPU到达同步屏障后完成全部同步。在大型GPU集群中，StragglAR相较于主流带宽高效算法实现2倍理论加速，突破了带宽最优同步AllReduce的下界。在8卡服务器上，相比最先进的AllReduce算法提速25%。
原文摘要 · Abstract (English)
Distributed machine learning workloads use data and tensor parallelism for training and inference, both of which rely on the AllReduce collective to synchronize gradients or activations. However, AllReduce algorithms are delayed by the slowest GPU to reach the synchronization barrier before the collective (i.e., the straggler). To address this challenge, we propose StragglAR: a parallel algorithm for AllReduce that accelerates distributed training and inference by exploiting natural variation in GPU execution times. StragglAR implements a ReduceScatter among the remaining GPUs during the straggler-induced delay, and then executes a novel collective algorithm to complete the AllReduce once the final GPU reaches the synchronization barrier. StragglAR achieves a 2x theoretical speedup over popular bandwidth-efficient algorithms for large GPU clusters, surpassing the lower bound for bandwidth-optimal synchronous AllReduce by leveraging the asymmetry in when GPUs reach the synchronization barrier. On an 8-GPU server, StragglAR provides a 25% speedup over state-of-the-art AllReduce algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。