arXiv:2507.08784cs.LGmath.OC2025-07被引 13

提出首个有收敛保证的低秩梯度压缩方法,提升分布式学习通信效率。

Greedy Low-Rank Gradient Compression for Distributed Learning with Convergence Guarantees

  • 采用贪婪策略选最相关信息子空间,结合误差反馈修正偏差。
  • 理论证明收敛速率达到 $\mathcal{O}(σ/\sqrt{NT} + 1/T)$,实现线性加速。
  • 适合大规模分布式训练场景,尤其对通信受限任务有显著优势。

分布式优化在大规模信号处理与机器学习中至关重要,但通信开销仍是主要瓶颈。低秩梯度压缩通过用低秩矩阵近似传输梯度以减少通信量,是一种有前景的解决方案。现有方法通常采用随机或贪婪压缩策略:随机方法将梯度投影到随机子空间,引入高方差,降低实际性能;贪婪方法选择最具信息量的子空间,表现优异但缺乏收敛保证。为填补这一空白,我们提出 GreedyLore——首个具有严格收敛保证的贪婪低秩梯度压缩算法。GreedyLore 引入误差反馈以纠正贪婪压缩带来的偏差,并设计半懒惰子空间更新机制,确保压缩算子在所有迭代中保持收缩性。理论上,该方法在 MSGD 与 Adam 等标准优化器下达到 $\mathcal{O}(σ/\sqrt{NT} + 1/T)$ 的收敛速率,首次实现低秩梯度压缩的线性加速收敛。大量实验验证了理论结果。

原文摘要 · Abstract (English)

Distributed optimization is pivotal for large-scale signal processing and machine learning, yet communication overhead remains a major bottleneck. Low-rank gradient compression, in which the transmitted gradients are approximated by low-rank matrices to reduce communication, offers a promising remedy. Existing methods typically adopt either randomized or greedy compression strategies: randomized approaches project gradients onto randomly chosen subspaces, introducing high variance and degrading empirical performance; greedy methods select the most informative subspaces, achieving strong empirical results but lacking convergence guarantees. To address this gap, we propose GreedyLore--the first Greedy Low-Rank gradient compression algorithm for distributed learning with rigorous convergence guarantees. GreedyLore incorporates error feedback to correct the bias introduced by greedy compression and introduces a semi-lazy subspace update that ensures the compression operator remains contractive throughout all iterations. With these techniques, we prove that GreedyLore achieves a convergence rate of $\mathcal{O}(σ/\sqrt{NT} + 1/T)$ under standard optimizers such as MSGD and Adam--marking the first linear speedup convergence rate for low-rank gradient compression. Extensive experiments are conducted to validate our theoretical findings.

分布式学习梯度压缩收敛分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。