提出新框架，让分布式训练通信开销不再随样本数增长
Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learning
- 结合本地更新与方差缩减，降低局部噪声影响
- 通信轮次复杂度仅与参与设备数相关，不再依赖总样本数
- 适合大规模分布式训练，尤其在设备数远小于样本数时优势明显
通信开销是可扩展分布式学习的关键瓶颈。现有方法如Local SGD、Minibatch SGD及其加速变体虽能高效利用数据点，但其通信轮次复杂度仍随总样本数 $N$ 增长。本文提出Local MixVR，一种融合本地更新与方差缩减技术的分布式框架，首次消除通信复杂度对 $N$ 的依赖，复杂度仅与工作节点数 $M$ 相关。在 $M < O(N^{1/4})$ 的常见场景下，Local MixVR优于当前最优的Minibatch Accelerated SGD基线，填补了分布式优化中的长期空白，建立了通信高效训练的新范式。
原文摘要 · Abstract (English)
Communication overhead is a crucial bottleneck in scalable distributed learning. While existing methods aim to efficiently utilize data points, such as Local SGD, Minibatch SGD, and their accelerated variants, they still exhibit communication-round complexity that scales with the total number of samples $N$. In this paper, we introduce Local MixVR, a distributed framework that integrates local updates with variance-reduction techniques to mitigate local noise. We show that Local MixVR is the first distributed method to eliminate the dependence of communication complexity on $N$, achieving a complexity that scales only with the number of workers $M$. In common regimes where $M<O\left(N^{1/4}\right)$, Local MixVR outperforms the state-of-the-art Minibatch Accelerated SGD baseline, bridging a long-standing gap in distributed optimization and establishing a new paradigm for communication-efficient training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。