通过过滤冲突梯度，提升分布式训练的泛化能力
Beyond Gradient Averaging in Parallel Optimization: Improved Robustness through Gradient Agreement Filtering
- 计算微批次间梯度夹角，剔除冲突更新再平均
- 小批量下验证准确率提升最多18.2%，训练更稳定
- 适合资源受限场景，显著降低计算开销
我们提出梯度一致性过滤（GAF），改进分布式深度学习中的梯度平均方法。传统数据并行随机梯度下降在训练后期常出现微批次梯度正交或负相关，导致过拟合和泛化能力下降。本文通过计算微梯度间的余弦距离，在平均前过滤冲突更新，有效降低梯度方差。实验表明，在CIFAR-100和CIFAR-100N-Fine等标准图像分类任务上，该方法可将验证准确率提升最多18.2%，同时支持极小微批量（如1）训练而不失稳定性，使计算量减少近一个数量级。
原文摘要 · Abstract (English)
We introduce Gradient Agreement Filtering (GAF) to improve on gradient averaging in distributed deep learning optimization. Traditional distributed data-parallel stochastic gradient descent involves averaging gradients of microbatches to calculate a macrobatch gradient that is then used to update model parameters. We find that gradients across microbatches are often orthogonal or negatively correlated, especially in late stages of training, which leads to memorization of the training set, reducing generalization. In this paper, we introduce a simple, computationally effective way to reduce gradient variance by computing the cosine distance between micro-gradients during training and filtering out conflicting updates prior to averaging. We improve validation accuracy with significantly smaller microbatch sizes. We also show this reduces memorizing noisy labels. We demonstrate the effectiveness of this technique on standard image classification benchmarks including CIFAR-100 and CIFAR-100N-Fine. We show this technique consistently outperforms validation accuracy, in some cases by up to 18.2\% compared to traditional training approaches while reducing the computation required nearly an order of magnitude because we can now rely on smaller microbatch sizes without destabilizing training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。