通过贝叶斯推断优化梯度压缩，提升分布式训练精度
Novel Gradient Sparsification Algorithm via Bayesian Inference
- 将梯度稀疏化建模为贝叶斯推断问题，用后验估计确定最优压缩掩码
- 在CIFAR-10上以0.1%稀疏度训练ResNet-18，比标准Top-k高8%准确率
- 适合需要高效通信的分布式深度学习场景，尤其关注收敛稳定性
误差累积是分布式梯度下降中Top-$k$稀疏化方法的核心机制，它隐式缩放学习率，防止横向移动变慢，但也可能损害收敛性。本文提出一种新型稀疏化算法——正则化Top-$k$（RegTop-$k$），用于控制误差累积带来的学习率缩放。该算法将梯度稀疏化视为一个推断问题，通过最大后验估计确定贝叶斯最优稀疏化掩码，并利用历史聚合梯度评估后验统计量，据此优先保留本地梯度分量。在CIFAR-10数据集上使用ResNet-18的数值实验表明，在0.1%稀疏度下，RegTop-$k$比标准Top-$k$提升约8%的准确率。
原文摘要 · Abstract (English)
Error accumulation is an essential component of the Top-$k$ sparsification method in distributed gradient descent. It implicitly scales the learning rate and prevents the slow-down of lateral movement, but it can also deteriorate convergence. This paper proposes a novel sparsification algorithm called regularized Top-$k$ (RegTop-$k$) that controls the learning rate scaling of error accumulation. The algorithm is developed by looking at the gradient sparsification as an inference problem and determining a Bayesian optimal sparsification mask via maximum-a-posteriori estimation. It utilizes past aggregated gradients to evaluate posterior statistics, based on which it prioritizes the local gradient entries. Numerical experiments with ResNet-18 on CIFAR-10 show that at $0.1\%$ sparsification, RegTop-$k$ achieves about $8\%$ higher accuracy than standard Top-$k$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。