通过优化权重的二阶统计量，提升模型对抗攻击的鲁棒性。
S2O: Enhancing Adversarial Training with Second-Order Statistics of Weights
- 将权重视为随机变量，基于二阶统计优化增强训练
- 理论证明可显著收紧鲁棒泛化边界，实验验证效果提升
- 适用于各类对抗训练方法，适合关注模型鲁棒性的研究者
对抗训练已成为提升深度神经网络鲁棒性的有效方法。传统方法将其建模为对模型权重与对抗扰动的极小极大优化问题，通常使用SGD等梯度下降法优化权重。本文提出一种新思路：将模型权重视为随机变量，引入权重的二阶统计量优化（S²O）。我们挑战并放松了以往PAC-Bayesian框架中权重统计独立的常见但不现实假设，由此导出更优的PAC-Bayesian鲁棒泛化界。理论分析表明，优化权重的二阶统计量可显著收紧该界。实验结果表明，S²O不仅在独立使用时能有效提升模型的鲁棒性与泛化能力，还能无缝增强其他先进对抗训练技术。代码已开源：https://github.com/Alexkael/S2O。
原文摘要 · Abstract (English)
Adversarial training has emerged as a highly effective way to improve the robustness of deep neural networks (DNNs). It is typically conceptualized as a min-max optimization problem over model weights and adversarial perturbations, where the weights are optimized using gradient descent methods, such as SGD. In this paper, we propose a novel approach by treating model weights as random variables, which paves the way for enhancing adversarial training through \textbf{S}econd-Order \textbf{S}tatistics \textbf{O}ptimization (S$^2$O) over model weights. We challenge and relax a prevalent, yet often unrealistic, assumption in prior PAC-Bayesian frameworks: the statistical independence of weights. From this relaxation, we derive an improved PAC-Bayesian robust generalization bound. Our theoretical developments suggest that optimizing the second-order statistics of weights can substantially tighten this bound. We complement this theoretical insight by conducting an extensive set of experiments that demonstrate that S$^2$O not only enhances the robustness and generalization of neural networks when used in isolation, but also seamlessly augments other state-of-the-art adversarial training techniques. The code is available at https://github.com/Alexkael/S2O.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。