arXiv:2601.10251cs.LGcs.AI2026-01

改进SAM方法,让模型更有效避开尖锐损失区域。

X-SAM: Boosting Sharpness-Aware Minimization with Dominant-Eigenvector Gradient Correction

  • 用海森矩阵主特征向量方向修正梯度,提升优化精度
  • 实验显示在ImageNet等数据集上显著提升泛化性能
  • 适合追求高泛化能力的深度学习研究者参考

Sharpness-Aware Minimization (SAM) 通过最小化参数邻域内的最坏情况扰动损失来提升模型泛化能力。然而,训练过程中其优化行为常与理论预期不符:尖锐和平坦区域都可能产生小的扰动损失,导致梯度仍指向尖锐区域,削弱了正则化效果。本文从谱和几何角度分析,提出以梯度与海森矩阵主特征向量夹角作为尖锐性度量。当该夹角≤90°时,SAM的正则化效果被削弱。为此,我们提出X-SAM,通过沿主特征向量进行正交分解显式校正梯度,实现对海森矩阵最大特征值的更直接、高效的正则化。理论证明了X-SAM的收敛性及更优泛化能力,大量实验验证了其在ImageNet、CIFAR-10等数据集上的理论与实际优势。

原文摘要 · Abstract (English)

Sharpness-Aware Minimization (SAM) aims to improve generalization by minimizing a worst-case perturbed loss over a small neighborhood of model parameters. However, during training, its optimization behavior does not always align with theoretical expectations, since both sharp and flat regions may yield a small perturbed loss. In such cases, the gradient may still point toward sharp regions, failing to achieve the intended effect of SAM. To address this issue, we investigate SAM from a spectral and geometric perspective: specifically, we utilize the angle between the gradient and the leading eigenvector of the Hessian as a measure of sharpness. Our analysis illustrates that when this angle is less than or equal to ninety degrees, the effect of SAM's sharpness regularization can be weakened. Furthermore, we propose an explicit eigenvector-aligned SAM (X-SAM), which corrects the gradient via orthogonal decomposition along the top eigenvector, enabling more direct and efficient regularization of the Hessian's maximum eigenvalue. We prove X-SAM's convergence and superior generalization, with extensive experimental evaluations confirming both theoretical and practical advantages.

优化算法泛化提升Hessian分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。