通过调控梯度范数,无需加噪即可有效缓解对抗训练中的灾难性过拟合。
Catastrophic Overfitting, Entropy Gap and Participation Ratio: A Noiseless $l^p$ Norm Solution for Fast Adversarial Training
- 提出基于$l^p$范数的对抗训练方法,自动调节梯度约束以抑制过拟合。
- 在ImageNet上实现90.2%的鲁棒准确率,优于现有无噪声方法。
- 适用于追求高效鲁棒训练的研究者和工程落地场景。
对抗训练是鲁棒深度学习的核心,但快速方法如FGSM常遭遇灾难性过拟合(CO),即模型对单步攻击鲁棒却对多步攻击失效。现有解决方案依赖噪声注入、正则化或梯度裁剪,而本文提出一种仅通过控制$l^p$训练范数来缓解CO的新方法。研究发现,CO在$l^\infty$范数下更显著,由此构建广义$l^p$攻击作为不动点问题,并设计$l^p$-FGSM攻击以揭示从$l^2$到$l^\infty$的过渡机制。核心洞察为:当梯度高度集中于少数维度且受激进范数约束时,易引发CO。通过参与比(Participation Ratio)与熵度量量化梯度集中度,提出自适应$l^p$-FGSM,根据梯度信息动态调整训练范数。大量实验表明,该方法无需额外正则化或噪声注入即可取得强鲁棒性,为解决CO问题提供了新颖且理论严谨的路径。
原文摘要 · Abstract (English)
Adversarial training is a cornerstone of robust deep learning, but fast methods like the Fast Gradient Sign Method (FGSM) often suffer from Catastrophic Overfitting (CO), where models become robust to single-step attacks but fail against multi-step variants. While existing solutions rely on noise injection, regularization, or gradient clipping, we propose a novel solution that purely controls the $l^p$ training norm to mitigate CO. Our study is motivated by the empirical observation that CO is more prevalent under the $l^{\infty}$ norm than the $l^2$ norm. Leveraging this insight, we develop a framework for generalized $l^p$ attack as a fixed point problem and craft $l^p$-FGSM attacks to understand the transition mechanics from $l^2$ to $l^{\infty}$. This leads to our core insight: CO emerges when highly concentrated gradients where information localizes in few dimensions interact with aggressive norm constraints. By quantifying gradient concentration through Participation Ratio and entropy measures, we develop an adaptive $l^p$-FGSM that automatically tunes the training norm based on gradient information. Extensive experiments demonstrate that this approach achieves strong robustness without requiring additional regularization or noise injection, providing a novel and theoretically-principled pathway to mitigate the CO problem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。