提出infoSAM算法，解决SAM模型退化问题并提升泛化性能。
Preconditioned Sharpness-Aware Minimization: Unifying Analysis and a Novel Learning Algorithm
- 用预条件技术统一多种SAM变体，理论分析更完整
- infoSAM通过噪声估计调整梯度，缓解对抗性退化问题
- 在多个基准上表现优于传统SAM，适合追求鲁棒性的研究者
为寻找损失函数曲面中平坦区域的解，尖锐感知最小化（SAM）已成为提升深度神经网络泛化能力的重要工具。尽管已有多种SAM变体被提出，但缺乏统一框架指导算法设计。本文引入预条件（pre）机制，统一现有SAM方法，不仅提供统一收敛性分析，还带来新洞见。基于preSAM，提出新型算法infoSAM，通过依据噪声估计调整梯度，有效缓解SAM中的对抗性模型退化问题。大量数值实验表明，infoSAM在多个基准测试中均表现出色。
原文摘要 · Abstract (English)
Targeting solutions over `flat' regions of the loss landscape, sharpness-aware minimization (SAM) has emerged as a powerful tool to improve generalizability of deep neural network based learning. While several SAM variants have been developed to this end, a unifying approach that also guides principled algorithm design has been elusive. This contribution leverages preconditioning (pre) to unify SAM variants and provide not only unifying convergence analysis, but also valuable insights. Building upon preSAM, a novel algorithm termed infoSAM is introduced to address the so-called adversarial model degradation issue in SAM by adjusting gradients depending on noise estimates. Extensive numerical tests demonstrate the superiority of infoSAM across various benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。