arXiv:2605.26929cs.LG2026-05被引 1

用正交优化器提升对抗训练鲁棒性,效果优于AdamW且不依赖显式权重收缩。

When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study

论文配图:When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study
图 1 · 摘自论文原文
  • 通过近似极分解实现矩阵更新正交化,控制梯度谱范数增长。
  • 在5种架构、3类攻击模型下,对齐甚至超越SGD表现,显著优于AdamW。
  • 适用于需高鲁棒性的深度学习系统,尤其适合视觉模型和强对抗环境。

对抗训练(AT)是抵御对抗攻击最可靠的实证防御方法之一,其鲁棒性高度依赖于底层极小极大目标的优化方式。实践中,随机梯度下降(SGD)仍是AT的默认优化器,而自适应优化器虽在标准训练中表现优异,却常导致鲁棒性下降。近期,Muon优化器通过近似极分解实现矩阵更新正交化,在大规模训练中表现出色,内存开销与SGD相当。本文研究:正交化优化能否在强且异构的威胁模型下提升AT?理论分析表明,Muon通过对矩阵更新施加谱范数稳定性上限,抑制了训练动态中的谱值无控增长,无需显式压缩权重。实验上,在5种架构及三种ℓ_p威胁模型(ℓ∞、ℓ₁、ℓ₂)及其并集下,Muon在CNN上与SGD性能相当,在CNN与ViT上均显著优于AdamW。结果表明,优化器几何结构是对抗训练中的关键安全因素,并明确了正交更新有益的实证范围。整体而言,优化器设计是对抗训练中不可忽视的安全核心。

原文摘要 · Abstract (English)

Adversarial training (AT) remains one of the most reliable empirical defenses against adversarial attacks. Its robustness critically depends on how the underlying min-max objective is optimized. In practice, Stochastic Gradient Descent (SGD) optimizer remains the default optimization choice for AT, whereas adaptive optimizers often improve standard training but may yield inferior robustness. Recently, the Muon optimizer, which orthogonalizes matrix-valued updates via an approximate polar decomposition, has achieved notable success in large-scale training at a memory cost comparable to SGD. This raises a security-relevant question: \textit{can orthogonalized optimization improve AT under strong and heterogeneous threat models?} Focusing on this problem, we conduct a comprehensive theoretical and empirical study. Theoretically, we show that Muon imposes a spectral-norm stability ceiling on matrix updates, limiting uncontrolled spectral growth in the training dynamics without explicitly shrinking the learned weights. Empirically, across five architectures and three $\ell_p$ threat models ($\ell_\infty$, $\ell_1$, $\ell_2$) and their union, Muon is competitive with SGD on CNNs and substantially outperforms AdamW on both CNNs and ViTs. These results identify optimizer geometry as a security-relevant factor in adversarial training, while clarifying the empirical regimes in which orthogonalized updates are beneficial. Overall, our findings highlight optimizer design as a security-critical component of AT.

对抗训练优化器设计正交化鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。