用杆流模型更精准模拟Adam在稳定边界上的优化过程。
A Rod Flow Model for Adam at the Edge of Stability
- 将杆流扩展到含动量的Adam，联合建模参数与一阶动量
- 在多种架构上，杆流比传统稳定流更准跟踪边缘稳定轨迹
- 适用于研究优化器动态，尤其适合关注训练稳定性的人
Cohen等人（arXiv:2207.14484）观察到自适应梯度方法如Adam在稳定边界运行。尽管已有大量关于梯度下降在稳定边界连续建模的研究，但将其扩展至动量类方法仍不充分。在梯度下降框架中，Regis等人（arXiv:2602.01480）提出杆流模型，将连续迭代视为一维延伸物体——“杆”。本文将杆流推广至Adam，通过在参数与一阶动量$(w, m)$的联合相空间中建模，并将二阶动量$ν$视为平滑辅助变量。同时，我们还为Heavy Ball、Nesterov动量，以及标量和分量级的RMSProp、Adam、NAdam构建了杆流模型。针对全部八种优化器，在典型机器学习架构上进行实验，结果表明，杆流在边缘稳定区间对离散迭代的追踪显著优于对应稳定流模型。
原文摘要 · Abstract (English)
Cohen et al. (arXiv:2207.14484) observed that adaptive gradient methods such as Adam operate at the edge of stability. While there has been significant work on continuous-time modeling of gradient descent at the edge of stability, extending these models to momentum methods remains underdeveloped. In the gradient descent setting, Regis et al. (arXiv:2602.01480) introduced rod flow, which models consecutive iterates as an extended one-dimensional object -- a "rod." Here we extend rod flow to Adam by working in the joint phase space of parameters and first moment $(w, m)$ and treating the second moment $ν$ as a smooth auxiliary variable. We also develop rod flows for heavy ball momentum, Nesterov momentum, and scalar and per-component versions of RMSProp, Adam, and NAdam. For all eight optimizers, we empirically evaluate rod flow on representative machine learning architectures, where it tracks the discrete iterates through the edge-of-stability regime significantly more accurately than the corresponding stable flow.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。