证明动量随机梯度下降能稳定泛化,打破其可能损害泛化的猜想。
Stochastic Gradient Descent with Momentum is Algorithmically Stable

- 提出统一的动量算法框架,涵盖Polyak与Nesterov两种形式。
- 在平滑凸问题上建立紧致的平均模型稳定性界,无需损失函数Lipschitz假设。
- 首次获得最优泛化误差界,适用于任意动量参数[0,1)。
动量随机梯度下降(SGDM)是机器学习中使用最广泛的优化算法之一。尽管其优化性质已有广泛研究,但关于其在未见数据上的泛化能力仍缺乏深入理解。尤其有观点认为,虽然动量可加速训练,但可能削弱泛化性能。本文通过算法稳定性视角,首次对SGDM进行全面的泛化分析。我们提出了一个广义的SGDM框架,涵盖Polyak与Nesterov两种动量机制,并为平滑凸问题建立了紧致的平均模型稳定性边界。关键结果包括:利用轨迹上的小优化误差、适用于任意动量参数区间[0,1),且不依赖损失函数的Lipschitz连续性假设。进一步推导出广义SGDM的优化误差界,并结合泛化分析,得到同时适用于Polyak与Nesterov动量的最优超额总体风险界。
原文摘要 · Abstract (English)
Stochastic gradient descent with momentum (SGDM) is one of the most widely used optimization algorithms in machine learning. While optimization properties of SGDM have been extensively studied in the literature, it remains insufficiently understood whether and when SGDM can generalize well to unseen data. In particular, it has been conjectured that while momentum accelerates training, it may degrade generalization. In this paper, we close this gap by developing a comprehensive generalization analysis of SGDM through the lens of algorithmic stability. More specifically, we introduce a generalized SGDM framework that encompasses both Polyak's and Nesterov's momentum schemes, and establish tight on-average model stability bounds for smooth and convex problems. Notably, the obtained bounds exploit small optimization error bounds along the trajectory, apply to any momentum parameter in the interval $[0, 1)$, and do not require the commonly assumed Lipschitzness of loss functions. We further derive optimization error bounds for the generalized SGDM, and combine them with our generalization analyses to obtain optimal excess population risk bounds for SGDM with both Polyak's and Nesterov's momentum.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。