提升小模型对抗鲁棒性蒸馏的优化过程,让学生更懂老师。
MMARD: Improving the Min-Max Optimization Process in Adversarial Robustness Distillation
- 内层用教师的鲁棒预测引导生成更贴近决策边界的对抗样本。
- 外层通过三角关系建模增强学生对自然与对抗场景的理解能力。
- 可无缝集成现有方法,实现在多个数据集上的顶尖性能。
对抗鲁棒性蒸馏(ARD)是一种利用预训练鲁棒教师模型提升小容量模型鲁棒性的有效方法,其本质为一个极小极大优化过程:内层生成对抗样本,外层训练学生模型。然而,现有方法存在两方面问题:内层生成的训练样本远离教师决策边界,导致重要鲁棒信息丢失;外层学生模型在自然与鲁棒场景中学习解耦,造成鲁棒性饱和,性能高度依赖教师选择。为此,本文提出通用的极小极大优化对抗鲁棒性蒸馏(MMARD)方法。内层引入教师的鲁棒预测,驱动对抗样本更接近教师决策边界,挖掘更多鲁棒知识;外层提出基于三角关系的结构化信息建模方法,衡量模型在自然与鲁棒场景中的互信息,强化其对多场景映射关系的理解。实验表明,MMARD在多个基准上达到当前最优性能,且具有即插即用特性,可方便地与现有方法结合。
原文摘要 · Abstract (English)
Adversarial Robustness Distillation (ARD) is a promising task to boost the robustness of small-capacity models with the guidance of the pre-trained robust teacher. The ARD can be summarized as a min-max optimization process, i.e., synthesizing adversarial examples (inner) & training the student (outer). Although competitive robustness performance, existing ARD methods still have issues. In the inner process, the synthetic training examples are far from the teacher's decision boundary leading to important robust information missing. In the outer process, the student model is decoupled from learning natural and robust scenarios, leading to the robustness saturation, i.e., student performance is highly susceptible to customized teacher selection. To tackle these issues, this paper proposes a general Min-Max optimization Adversarial Robustness Distillation (MMARD) method. For the inner process, we introduce the teacher's robust predictions, which drive the training examples closer to the teacher's decision boundary to explore more robust knowledge. For the outer process, we propose a structured information modeling method based on triangular relationships to measure the mutual information of the model in natural and robust scenarios and enhance the model's ability to understand multi-scenario mapping relationships. Experiments show our MMARD achieves state-of-the-art performance on multiple benchmarks. Besides, MMARD is plug-and-play and convenient to combine with existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。