用谱范数优化提升模型鲁棒性，效果优于传统方法。
Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm
- 在参数扰动中引入层间谱范数，更贴合权重矩阵结构。
- 结合Muon优化器的外层更新，在ImageNet上精度领先。
- 适合追求高鲁棒性和泛化能力的深度学习研究者。
Sharpness-Aware Minimization (SAM) 通过鼓励对小的最坏情况参数扰动不敏感来提升泛化能力。然而，'小'扰动的概念本质上依赖于几何结构：尽管现有 SAM 变体探索了多种选择，但实践中哪种几何结构最有效仍不明确。近期关于矩阵感知优化的工作，特别是 Muon 优化器，表明尊重隐藏层权重的矩阵结构可带来强劲的实证表现。受此启发，我们在 SAM 的两个阶段中均引入矩阵感知几何：对矩阵型隐藏层参数采用分层谱内扰动，并与 AdamW/SGDW 或 Muon 在外层更新中结合。在 ViT-Small/16 与 ResNet-50 的 ImageNet-1K 实验中，谱内扰动与 Muon 外层更新的组合表现持续优异，成为所评估方法中在两个模型上均取得最佳验证精度的方案。
原文摘要 · Abstract (English)
Sharpness-Aware Minimization (SAM) aims to improve generalization by encouraging insensitivity to small, worst-case parameter perturbations. However, the notion of a "small" perturbation is inherently geometry-dependent: while existing SAM variants have explored a wide range of choices, a clear perspective on which geometries are most effective in practice remains elusive. Recent work on matrix-aware optimization, particularly the Muon optimizer, suggests that respecting the matrix structure of hidden-layer weights can lead to strong empirical performance. Motivated by this, we study matrix-aware geometry in both stages of SAM: we introduce a layerwise spectral inner perturbation for matrix-valued hidden-layer parameters and combine it with either AdamW/SGDW or Muon in the outer update. Across ImageNet-1K experiments on ViT-Small/16 and ResNet-50, we find that the combination of a spectral inner step with a Muon outer step performs consistently strongly, achieving the best validation accuracy on both models among the evaluated methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。