改进SAM算法，让模型更关注关键方向的平坦性。
Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimization
- 通过分析海森矩阵最大特征值，揭示SAM训练机制
- 发现扰动方向与主特征向量对齐不足，限制性能提升
- 提出Eigen-SAM，显式对齐主特征向量提升泛化能力
Sharpness-Aware Minimization (SAM) 在多种任务中表现出优异的泛化能力，但其内在原理仍不清晰。本文利用海森矩阵的最大特征值作为尖锐度的度量，建立了一个三阶随机微分方程，揭示其训练动态由二阶和三阶项共同驱动。我们发现，扰动向量与最高特征向量的对齐程度是SAM有效正则化尖锐度的关键，但实际中该对齐常不充分，限制了其效率。基于此，我们提出Eigen-SAM，显式地通过将扰动向量对齐到海森矩阵的主特征向量来正则化最大特征值。通过大量实验验证了理论的有效性和所提方法的实际优势。代码已开源：https://github.com/RitianLuo/EigenSAM。
原文摘要 · Abstract (English)
Sharpness-Aware Minimization (SAM) has attracted significant attention for its effectiveness in improving generalization across various tasks. However, its underlying principles remain poorly understood. In this work, we analyze SAM's training dynamics using the maximum eigenvalue of the Hessian as a measure of sharpness, and propose a third-order stochastic differential equation (SDE), which reveals that the dynamics are driven by a complex mixture of second- and third-order terms. We show that alignment between the perturbation vector and the top eigenvector is crucial for SAM's effectiveness in regularizing sharpness, but find that this alignment is often inadequate in practice, limiting SAM's efficiency. Building on these insights, we introduce Eigen-SAM, an algorithm that explicitly aims to regularize the top Hessian eigenvalue by aligning the perturbation vector with the leading eigenvector. We validate the effectiveness of our theory and the practical advantages of our proposed approach through comprehensive experiments. Code is available at https://github.com/RitianLuo/EigenSAM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。