arXiv:2607.06151cs.LGmath.PR2026-07

用外梯度思想改进尖锐感知优化,提升模型泛化能力

Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning

论文配图:Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning
图 1 · 摘自论文原文
  • 引入外梯度机制,分两步更新以捕捉损失曲面几何
  • 在多个数据集上测试准确率超越SGD、Adam和SAM
  • 对扰动半径不敏感,调参更简单,适合实际部署

泛化仍是深度学习的核心挑战,传统优化器如随机梯度下降(SGD)常收敛至尖锐极小值,导致过拟合并降低在未见数据上的性能。基于尖锐感知最小化(SAM),我们提出一种受外梯度启发的新型优化器EISAM,通过外梯度技术增强泛化能力。EISAM采用两步更新:预测步骤分析损失曲面几何,扰动步骤结合基础优化器细化更新。该方法在多个基准数据集上的实验表明,其测试准确率和训练效率均优于SGD、Adam和SAM。理论分析进一步证明,EISAM通过引导参数进入曲率更低的平坦极小值,收紧了泛化界。配套的超参数分析提供了实用调参指导,使EISAM成为鲁棒、可扩展且广泛适用的优化方案。

原文摘要 · Abstract (English)

Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data. Building on Sharpness-Aware Minimization (SAM), for seeking flat minima associated with improved generalization, we propose the Extragradient-Inspired Sharpness-Aware Minimization (EISAM), a novel optimizer that enhances generalization via the extragradient technique. EISAM uses a two-step update process: a prediction step investigating the geometry of the loss landscape and a perturbation step that refines updates with a base optimizer. This approach achieves better generalization performance than SAM. Crucially, EISAM reduces sensitivity to the perturbation radius, enhancing robustness, and simplifying the tuning across diverse settings. Extensive experiments on benchmark datasets demonstrate that EISAM consistently outperforms SGD, Adaptive Moment Estimation (Adam), and SAM in test accuracy and training efficiency across various architectures. Theoretical analysis further confirms that EISAM tightens the generalization bound by steering parameters toward flatter minima with reduced curvature. Accompanied by a thorough hyperparameter analysis, EISAM offers practical tuning guidance, establishing it as a robust, scalable, and broadly applicable optimization solution that advances both the theory and practice in deep learning.

优化器泛化深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。