提出可兼容All-Reduce的Top-K压缩器，显著降低通信开销。
An All-Reduce Compatible Top-K Compressor for Communication-Efficient Distributed Learning
- 用轻量级梯度草图对齐各节点稀疏模式，实现无索引All-Reduce
- 在标准假设下达到线性加速，收敛速度优于原版EF21M
- 兼具Rand-K鲁棒性与Top-K性能，训练时间减少最多60.7%
通信仍是大规模分布式机器学习的核心瓶颈，梯度稀疏化成为缓解该问题的有前景策略。然而，现有梯度压缩器存在明显局限：随机选择的Rand-$K$会丢失结构信息，实际表现差；而保留重要分量的Top-$K$失去收缩性，且需代价高昂的All-Gather操作。本文提出ARC-Top-$K$，一种兼容All-Reduce的Top-$K$压缩器，通过轻量级梯度草图对齐各节点稀疏模式，实现无需索引的All-Reduce，同时保留全局重要信息。该方法在理论上是收缩的，结合动量误差反馈（EF21M）后，在标准假设下实现线性加速，收敛速度优于原始EF21M。实验表明，ARC-Top-$K$在保持与Top-$K$相当精度的同时，将墙钟训练时间最多减少60.7%，提供了一种高效且可扩展的解决方案，兼具Rand-$K$的鲁棒性与Top-$K$的优异性能。
原文摘要 · Abstract (English)
Communication remains a central bottleneck in large-scale distributed machine learning, and gradient sparsification has emerged as a promising strategy to alleviate this challenge. However, existing gradient compressors face notable limitations: Rand-$K$ discards structural information and performs poorly in practice, while Top-$K$ preserves informative entries but loses the contraction property and requires costly All-Gather operations. In this paper, we propose ARC-Top-$K$, an {All-Reduce}-Compatible Top-$K$ compressor that aligns sparsity patterns across nodes using a lightweight sketch of the gradient, enabling index-free All-Reduce while preserving globally significant information. ARC-Top-$K$ is provably contractive and, when combined with momentum error feedback (EF21M), achieves linear speedup and sharper convergence rates than the original EF21M under standard assumptions. Empirically, ARC-Top-$K$ matches the accuracy of Top-$K$ while reducing wall-clock training time by up to 60.7\%, offering an efficient and scalable solution that combines the robustness of Rand-$K$ with the strong performance of Top-$K$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。