改进了SAM的优化方法，让模型找到更平坦的极小值点，提升泛化性能。
Tilted Sharpness-Aware Minimization
- 通过指数倾斜思想设计新优化器，更关注高损失区域的局部解
- 在图像与文本任务中测试表现优于SAM和ERM，达到更平坦的极小值
- 算法基于哈密顿动力学离散化，优化过程更平滑稳定
Sharpness-Aware Minimization (SAM) 通过在损失曲面的邻域内寻找最大损失对应的参数，以搜索平坦极小值，从而提升过参数化模型的泛化能力。然而，这种极小-极大公式在高度非凸问题中计算困难，且仅关注最坏情况的局部解，可能忽略其他潜在有利解。本文提出倾斜式SAM（TSAM），一种受指数倾斜启发的平滑推广形式，通过倾斜超参数 $t$ 显式赋予高损失局部解更高优先级。当 $t \to \infty$ 时，TSAM 退化为 SAM。我们证明了 TSAM 比 SAM 更平滑，更易优化，并明确倾向于平坦极小值。基于哈密顿动力学离散化设计求解算法。实验表明，TSAM 能到达更平坦的局部极小值，在多种图像与文本任务上均显著优于 SAM 和 ERM 的测试性能。
原文摘要 · Abstract (English)
Sharpness-Aware Minimization (SAM) has been demonstrated to improve the generalization performance of overparameterized models by seeking flat minima on the loss landscape through optimizing model parameters that incur the largest loss within a neighborhood. Nevertheless, such min-max formulations are computationally challenging especially when the problem is highly non-convex. Additionally, focusing only on the worst-case local solution while ignoring potentially many other local solutions may be suboptimal when searching for flat minima. In this work, we propose Tilted SAM (TSAM), a smoothed generalization of SAM inspired by exponential tilting that effectively assigns higher priority to local solutions that incur larger losses. TSAM is parameterized by a tilt hyperparameter $t$ and reduces to SAM as $t$ approaches infinity. We show that TSAM is smoother than SAM and thus easier to optimize, and it explicitly favors flatter minima. We develop algorithms motivated by the discretization of Hamiltonian dynamics to solve TSAM. Empirically, TSAM arrives at flatter local minima and results in superior test performance than the baselines of SAM and ERM across a range of image and text tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。