分析了SGD+EMA在线性回归中的表现，揭示其为何更稳定且收敛更快。
Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression
- 用指数移动平均优化SGD，降低模型方差
- 偏差误差在数据协方差各特征子空间中呈指数衰减
- 为多种平均策略提供通用分析方法，适合研究者参考
指数移动平均（EMA）近年来在训练现代深度学习模型中广受青睐，尤其在基于扩散的生成模型中。然而，关于EMA有效性的理论解释仍十分有限。本文以高维线性回归这一最简单的过参数化学习任务为研究对象，建立了带有EMA的在线SGD的风险界。结果表明：(i) 带EMA的SGD的方差误差始终小于不使用平均的SGD；(ii) 与从一开始就进行迭代平均的SGD不同，带EMA的偏差误差在数据协方差矩阵的每个特征子空间中均呈指数衰减。此外，本文还发展出可推广至多种平均策略的分析技术。
原文摘要 · Abstract (English)
Exponential moving average (EMA) has recently gained significant popularity in training modern deep learning models, especially diffusion-based generative models. However, there have been few theoretical results explaining the effectiveness of EMA. In this paper, to better understand EMA, we establish the risk bound of online SGD with EMA for high-dimensional linear regression, one of the simplest overparameterized learning tasks that shares similarities with neural networks. Our results indicate that (i) the variance error of SGD with EMA is always smaller than that of SGD without averaging, and (ii) unlike SGD with iterate averaging from the beginning, the bias error of SGD with EMA decays exponentially in every eigen-subspace of the data covariance matrix. Additionally, we develop proof techniques applicable to the analysis of a broad class of averaging schemes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。