提出VASSO方法，解决SAM过拟合平坦谷问题，提升模型泛化能力。
VASSO: Variance Suppression for Sharpness-Aware Minimization
- 通过方差抑制稳定对抗扰动，防止SAM出现‘友好的’干扰
- 在视觉与语言任务上验证，结合SAM后泛化性能显著提升
- 适配高效SAM变体，兼顾计算效率与模型表现
Sharpness-aware minimization (SAM) 在提升深度神经网络泛化能力方面有明确优势。其通过考虑损失函数几何中的尖锐性，寻找‘平坦极小值’区域，即在邻域内最小化对抗扰动引发的最大损失。尽管尖锐性建模至关重要，但实际应用中SAM常受‘过度友好’的对抗扰动影响，限制了泛化上限。本文提出方差抑制（VASSO），一种可证明稳定的对抗扰动机制。将VASSO与SAM结合，在广泛的视觉与语言任务中数值验证了更强的泛化能力。当应用于计算高效的SAM变体时，VASSO实现了理想的泛化-计算权衡。
原文摘要 · Abstract (English)
Sharpness-aware minimization (SAM) has well-documented merits in enhancing generalization of deep neural network models. Accounting for sharpness in the loss function geometry, where neighborhoods of `flat minima' heighten generalization ability, SAM seeks `flat valleys' by minimizing the maximum loss provoked by an adversarial perturbation within the neighborhood. Although critical to account for sharpness of the loss function, in practice SAM suffers from `over-friendly adversaries,' which can curtail the outmost level of generalization. To avoid such `friendliness,' the present contribution fosters stabilization of adversaries through variance suppression (VASSO). VASSO offers a general approach to provably stabilize adversaries. In particular, when integrating VASSO with SAM, improved generalizability is numerically validated on extensive vision and language tasks. Once applied on top of a computationally efficient SAM variant, VASSO offers a desirable generalization-computation tradeoff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。