用双教师蒸馏提升模型抗攻击能力，同时不牺牲干净数据准确率。
Improving the Robustness/Accuracy Tradeoff Against Adversarial Attacks Using Information Bottleneck Distillation Through Dual Teachers
- 引入干净样本训练的教师模型，与鲁棒教师协同蒸馏。
- 在CIFAR-10/100上相比原方法提升干净数据准确率，对抗攻击精度相当。
- 适合关注鲁棒性与准确率平衡的研究者和工程应用。
深度神经网络在经典机器学习任务中表现卓越，但对对抗攻击敏感。现有对策如信息瓶颈蒸馏（IBD）虽提升鲁棒性，却降低干净输入准确率。本文扩展IBD框架，引入仅用干净数据训练的教师模型，与经过对抗训练的鲁棒教师共同指导学生模型。通过跨层注意力矩阵融合两者特征。在CIFAR-10和CIFAR-100上的实验表明，该方法在保持对抗样本精度的同时，显著提升干净样本分类准确率。性能优于原始IBD，且在清洁与鲁棒准确率的调和均值上媲美当前先进方法，如B-MTARD。还分析了不同训练设置对注意力模块的影响。
原文摘要 · Abstract (English)
Deep neural networks (DNNs) have achieved remarkable success in classical machine learning problems. However, they are known to be vulnerable to adversarial attacks. Countermeasures proposed in the literature, notably Information Bottleneck Distillation (IBD) introduced by Kuang et al., degrade the classification accuracy on clean inputs while improving the robustness to adversarial inputs. In this work, we extend the IBD framework by introducing an extra teacher model (clean teacher) trained with only clean inputs, into the distillation process from a robust teacher model trained by adversarial training. The features of both clean and robust teachers are transferred to the student through a cross-layer attention matrix. Experimental results on the CIFAR-10 and CIFAR-100 datasets show that the proposed method improves classification accuracy on clean samples compared to the original IBD, while maintaining similar accuracy on adversarial samples. Furthermore, our methods are competitive with state-of-the-art approaches, including the recent dual-teacher distillation framework B-MTARD, particularly in terms of the harmonic mean between clean and robust accuracy. We also analyze the impact of different training settings that have different influences on the attention module.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。