揭示动量梯度下降的隐含损失机制，给出精确逼近理论。
Modified Loss of Momentum Gradient Descent: Fine-Grained Analysis
- 将动量法视为带修正损失的普通梯度下降，实现数学等价
- 证明任意阶精度可达O(h^R)，R≥2，适用于全批量和小批量
- 发现隐藏在动量中的欧拉和纳拉亚纳多项式，揭示算法内在结构
我们分析了具有固定动量参数β∈(0,1)的Polyak动量梯度下降（HB），其记忆呈指数衰减。基于Kovachki与Stuart（2021）的工作，我们证明当步长h足够小时，在一个指数吸引的不变流形上，该算法等价于带有修正损失的普通梯度下降。尽管修正损失无显式表达式，我们可任意精度描述它，并证明全局（有限“时间”范围）近似误差为O(h^R)，对任意有限阶R≥2成立。随后，我们对HB的记忆无依赖近似背后的组合结构进行精细分析，发现其中隐藏着丰富的β的多项式族，包含欧拉多项式与纳拉亚纳多项式。我们推导出任意阶精度的连续修正方程（并附严格界），以及近似HB动态的主流形，推广了Rosca等（2023）的结果。近似定理涵盖全批量与小批量情形。这些理论成果为动量梯度下降的核心特性提供了新视角，并为其他优化算法的类似分析指明了路径。
原文摘要 · Abstract (English)
We analyze gradient descent with Polyak heavy-ball momentum (HB) whose fixed momentum parameter $β\in (0, 1)$ provides exponential decay of memory. Building on Kovachki and Stuart (2021), we prove that on an exponentially attractive invariant manifold the algorithm is exactly plain gradient descent with a modified loss, provided that the step size $h$ is small enough. Although the modified loss does not admit a closed-form expression, we describe it with arbitrary precision and prove global (finite "time" horizon) approximation bounds $O(h^{R})$ for any finite order $R \geq 2$. We then conduct a fine-grained analysis of the combinatorics underlying the memoryless approximations of HB, in particular, finding a rich family of polynomials in $β$ hidden inside which contains Eulerian and Narayana polynomials. We derive continuous modified equations of arbitrary approximation order (with rigorous bounds) and the principal flow that approximates the HB dynamics, generalizing Rosca et al. (2023). Approximation theorems cover both full-batch and mini-batch HB. Our theoretical results shed new light on the main features of gradient descent with heavy-ball momentum, and outline a road-map for similar analysis of other optimization algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。