提出新型抗拜占庭攻击的高效通信压缩算法，收敛速度更快且无需大批次。
Accelerating Byzantine-Robust Distributed Learning with Compressed Communication via Double Momentum and Variance Reduction
- 采用双动量梯度估计器结合误差反馈，提升鲁棒性与通信效率。
- 理论证明在ε-驻点下仅需O(ε⁻⁴)次迭代，加速版本为O(ε⁻³)。
- 适合大规模分布式训练场景，尤其适用于存在故障节点的系统。
在协同分布式学习中，拜占庭鲁棒性是优化算法的重要特性。此类算法常伴随大量参数传输，因此通信压缩至关重要。本文提出 Byz-DM21，一种新型抗拜占庭且通信高效的随机分布式学习算法。核心创新在于基于双动量机制的梯度估计器，融合近期误差反馈技术。利用该估计器，设计了标准与加速算法，在无需大批次的情况下仍保持对拜占庭工作者的鲁棒性。理论证明 Byz-DM21 可在 $\mathcal{O}(\varepsilon^{-4})$ 次迭代内收敛至 $\varepsilon$-驻点。为进一步提升效率，引入分布式变体 Byz-VR-DM21，通过各节点局部方差缩减逐步消除随机近似带来的方差。我们证明 Byz-VR-DM21 可在 $\mathcal{O}(\varepsilon^{-3})$ 次迭代内收敛至 $\varepsilon$-驻点。此外，结果扩展至满足 Polyak-Łojasiewicz 条件的情形。数值实验验证了方法的有效性。
原文摘要 · Abstract (English)
In collaborative and distributed learning, Byzantine robustness reflects a major facet of optimization algorithms. Such distributed algorithms are often accompanied by transmitting a large number of parameters, so communication compression is essential for an effective solution. In this paper, we propose Byz-DM21, a novel Byzantine-robust and communication-efficient stochastic distributed learning algorithm. Our key innovation is a novel gradient estimator based on a double-momentum mechanism, integrating recent advancements in error feedback techniques. Using this estimator, we design both standard and accelerated algorithms that eliminate the need for large batch sizes while maintaining robustness against Byzantine workers. We prove that the Byz-DM21 algorithm has a smaller neighborhood size and converges to $\varepsilon$-stationary points in $\mathcal{O}(\varepsilon^{-4})$ iterations. To further enhance efficiency, we introduce a distributed variant called Byz-VR-DM21, which incorporates local variance reduction at each node to progressively eliminate variance from random approximations. We show that Byz-VR-DM21 provably converges to $\varepsilon$-stationary points in $\mathcal{O}(\varepsilon^{-3 })$ iterations. Additionally, we extend our results to the case where the functions satisfy the Polyak-Łojasiewicz condition. Finally, numerical experiments demonstrate the effectiveness of the proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。