arXiv:2605.16134cs.LGcs.AI2026-05

用几何感知的优化方法,让模型避开局部尖峰陷阱。

Navigating Potholes with Geometry-Aware Sharpness Minimization

  • 结合二阶方法构建慢速几何预处理器,捕捉损失曲面宏观结构。
  • 在预处理器基础上进行快速尖锐度探测,提升逃离局部极小的能力。
  • 适用于视觉与序列建模,尤其擅长稳定训练过程中的优化路径。

Sharpness-aware minimization (SAM) 通过沿损失曲率高的方向扰动参数来寻找平坦的极小值,但忽略了损失函数的内在几何结构。本文提出 LLQR+SAM,将 SAM 与最近提出的 LLQR 框架所获得的可学习预处理器相结合。该预处理器以稀疏方式更新,并采用慢速指数移动平均,从而捕捉到损失曲面的平滑、低分辨率几何图像。随后的 SAM 扰动在此学习到的几何基础上进行,以更快的时间尺度探测曲率。理论分析表明,该双时间尺度结构并非仅出于计算便利:预处理器能增强在平均几何下平坦但在局部尖锐(如凹坑)的方向上的逃离信号。而宽广平坦的基底则保持稳定。实验表明,LLQR+SAM 在标准视觉与序列建模范畴上持续优于 SAM 和单独的 LLQR,支持了慢速学习几何与快速尖锐修正具有真正互补性的观点。

原文摘要 · Abstract (English)

Sharpness-aware minimization (SAM) encourages flat minima by perturbing parameters along directions of high loss curvature, but treats all parameter directions uniformly, ignoring the underlying loss geometry. We introduce LLQR+SAM, which combines SAM with a learned preconditioner obtained from the recently proposed LLQR framework, a second-order method that recasts steepest descent as a layerwise linear-quadratic regulator problem. The preconditioner is updated sparsely and maintained as a slow exponential moving average, so it captures a smoothed, low-resolution picture of the loss landscape geometry. The SAM perturbation then operates on top of this learned geometry, probing curvature at a faster timescale. We show that this two-timescale structure is not merely a computational convenience: theoretically, the preconditioner amplifies the SAM escape signal in directions that are flat under the average geometry but locally sharp (potholes). Wide, flat basins, by contrast, remain stable. Empirically, LLQR+SAM gives consistent gains over both SAM and LLQR alone across standard vision and sequence modeling benchmarks, supporting the view that slow learned geometry and fast sharpness correction are genuinely complementary.

优化算法几何感知深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。