提出一种抗故障且通信高效的分布式学习方法,收敛更精确。
Byzantine-Robust and Communication-Efficient Distributed Learning via Compressed Momentum Filtering
- 用动量滤波压缩梯度,抵御恶意节点干扰
- 无需限定批量大小,收敛至更小邻域,逼近理论极限
- 适合大规模私有数据训练,兼顾鲁棒性与效率
分布式学习已成为在私有数据孤岛间训练大规模机器学习模型的标准方法。尽管其提升了隐私保护和训练效率,但仍面临拜占庭鲁棒性和通信开销两大挑战。现有鲁棒且高效的方法通常在每轮或特定轮次依赖完整梯度信息,且仅收敛到比必要更大的解附近邻域。针对这些问题,本文提出一种新型随机分布式学习方法,无需对批量大小做要求,收敛至比所有已有方法更小的邻域,与理论下界一致。核心创新在于利用Polyak动量缓解压缩器偏差和随机梯度带来的噪声,从而在信息压缩下抵御拜占庭工作者。我们为非凸平滑损失函数下的算法提供了紧致复杂度上界证明,表明这些界在无拜占庭场景下与理论下界一致。最后,通过大量实验验证了该算法在二分类和图像分类任务上的实际有效性。
原文摘要 · Abstract (English)
Distributed learning has become the standard approach for training large-scale machine learning models across private data silos. While distributed learning enhances privacy preservation and training efficiency, it faces critical challenges related to Byzantine robustness and communication reduction. Existing Byzantine-robust and communication-efficient methods rely on full gradient information either at every iteration or at certain iterations with a probability, and they only converge to an unnecessarily large neighborhood around the solution. Motivated by these issues, we propose a novel Byzantine-robust and communication-efficient stochastic distributed learning method that imposes no requirements on batch size and converges to a smaller neighborhood around the optimal solution than all existing methods, aligning with the theoretical lower bound. Our key innovation is leveraging Polyak Momentum to mitigate the noise caused by both biased compressors and stochastic gradients, thus defending against Byzantine workers under information compression. We provide proof of tight complexity bounds for our algorithm in the context of non-convex smooth loss functions, demonstrating that these bounds match the lower bounds in Byzantine-free scenarios. Finally, we validate the practical significance of our algorithm through an extensive series of experiments, benchmarking its performance on both binary classification and image classification tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。