用指数移动平均优化模型权重，提升泛化与鲁棒性。
Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits
- 对梯度下降权重做指数移动平均，降低训练噪声。
- 相比普通SGD，EMA在噪声标签、预测一致性上表现更优。
- 无需大幅调整学习率，适合直接接入现有训练流程。
权重平均是深度学习中常用的技术，常用于提升模型泛化能力或作为教师模型。本文系统研究了权重的指数移动平均（EMA）机制。通过分析其训练动态，提出超参数调优建议，并发现EMA在早期即表现良好，部分解释了其作为教师模型的成功。实验表明，由于平均过程天然抑制噪声，EMA可减少学习率衰减的需求，实现隐式正则化。对比最终迭代解，EMA解不仅泛化能力更强，还显著提升对噪声标签的鲁棒性、预测一致性、校准度及迁移性能。因此，我们建议将权重的EMA作为简单有效的插件，直接提升深度模型性能。
原文摘要 · Abstract (English)
Weight averaging of Stochastic Gradient Descent (SGD) iterates is a popular method for training deep learning models. While it is often used as part of complex training pipelines to improve generalization or serve as a `teacher' model, weight averaging lacks proper evaluation on its own. In this work, we present a systematic study of the Exponential Moving Average (EMA) of weights. We first explore the training dynamics of EMA, give guidelines for hyperparameter tuning, and highlight its good early performance, partly explaining its success as a teacher. We also observe that EMA requires less learning rate decay compared to SGD since averaging naturally reduces noise, introducing a form of implicit regularization. Through extensive experiments, we show that EMA solutions differ from last-iterate solutions. EMA models not only generalize better but also exhibit improved i) robustness to noisy labels, ii) prediction consistency, iii) calibration and iv) transfer learning. Therefore, we suggest that an EMA of weights is a simple yet effective plug-in to improve the performance of deep learning models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。