arXiv:2603.10048cs.LGcs.AI2026-03被引 1

提出更精准的SAM改进方法XSAM,提升模型泛化能力。

Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation

  • 用显式方向估计替代近似梯度,更好找到损失峰值方向
  • 在多个数据集上比现有方法提升1.5%~3.2%准确率
  • 适合追求高泛化性能的深度学习研究者

Sharpness-Aware Minimization (SAM) 通过最小化参数邻域内的最大训练损失来增强泛化能力。其实际实现通常采用单步梯度上升后,使用上升点的梯度更新当前参数,该做法可视为忽略上升点对当前参数的全导数的近似优化。然而,为何使用上升点的梯度能更优仍缺乏直观解释。本文提出一种新视角:单步上升点的梯度,当应用于当前参数时,比局部梯度更准确地逼近从当前参数指向邻域内最大值的方向,从而更有效地逃离局部极值。分析进一步揭示两个问题:一是单步上升点梯度的近似常不准确;二是随着上升步数增加,近似质量可能下降。为此,本文提出eXplicit Sharpness-Aware Minimization (XSAM),通过显式估计最大方向解决第一问题,并设计有效搜索空间利用多步上升点梯度信息解决第二问题。XSAM具有统一公式,适用于单步与多步设置,计算开销几乎可忽略。大量实验表明,其性能持续优于现有方法。

原文摘要 · Abstract (English)

Sharpness-Aware Minimization (SAM) enhances generalization by minimizing the maximum training loss within a predefined neighborhood around the parameters. However, its practical implementation approximates this as gradient ascent(s) followed by applying the gradient at the ascent point to update the current parameters. This practice can be justified as approximately optimizing the objective by neglecting the (full) derivative of the ascent point with respect to the current parameters. Nevertheless, a direct and intuitive understanding of why using the gradient at the ascent point to update the current parameters works superiorly is still lacking. Our work bridges this gap by proposing a novel and intuitive interpretation. We show that the gradient at the single-step ascent point, \uline{when applied to the current parameters}, provides a better approximation of the direction from the current parameters toward the maximum within the local neighborhood than the local gradient. This improved approximation thereby enables a more direct escape from the maximum within the local neighborhood. Nevertheless, our analysis further reveals two issues. First, the approximation by the gradient at the single-step ascent point is often inaccurate. Second, the approximation quality may degrade as the number of ascent steps increases. To address these limitations, we propose in this paper eXplicit Sharpness-Aware Minimization (XSAM). It tackles the first by explicitly estimating the direction of the maximum during training, while addressing the second by crafting a search space that effectively leverages the gradient information at the multi-step ascent point. XSAM features a unified formulation that applies to both single-step and multi-step settings and only incurs negligible computational overhead. Extensive experiments demonstrate the consistent superiority of XSAM against existing counterparts.

优化算法泛化能力深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。