揭示优化器在稳定边缘的振荡行为与离散力学的深层联系。
Variational Structure at the Edge of Stability
- 提出扩展的边耦合函数，统一描述梯度下降、动量及Nesterov动量的周期轨道。
- 证明边耦合的临界点对应不动点和双周期轨道，海森矩阵决定其稳定性。
- 发现边耦合等价于对称Verlet作用量，建立优化与离散力学的数学桥梁。
当离散时间优化器运行在稳定边缘时，表现出近似双周期振荡行为，类似保辛系统（如辛积分器生成的动力学）。然而，离散优化器在稳定边缘与离散力学之间的精确关联仍待深入探索。近期，Litman提出了“边耦合”：一种定义在连续梯度下降迭代点上的泛函，其临界点编码了梯度下降动力学的不动点与二周期轨道。本文将边耦合拓展至重球法（heavy-ball）与Nesterov动量方法。我们证明其临界点可表征不动点与双周期轨道，且其海森矩阵刻画了这些轨道的稳定性。此外，我们发现边耦合可被识别为对称Verlet作用量，从而形式化了稳定边缘与离散力学之间的联系。
原文摘要 · Abstract (English)
When discrete-time optimizers operate at the edge of stability, they exhibit near-two-periodic behavior. These oscillatory dynamics are reminiscent of conservative systems, such as the dynamics generated by symplectic integrators. However, a precise formulation of the connection between discrete-time optimizers at the edge of stability and discrete mechanics remains underexplored. Recently, Litman introduced the "edge coupling": a functional on consecutive gradient descent iterates whose critical points encode the fixed points and two-point orbits of the gradient descent dynamics. Here we extend the edge coupling to heavy-ball and Nesterov momentum. We show that its critical points characterize the fixed points and two-point orbits, with its Hessian characterizing their stability. We also show that the edge coupling can be identified with the symmetric Verlet action, formalizing the connection between the edge of stability and discrete mechanics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。