发现鲁棒教师反而让学生变差的原因:教得太自信会误导学生记错噪声。
Toward Understanding Adversarial Distillation: Why Robust Teachers Fail

- 分析师生在不可学习样本上的信心不匹配问题
- 实验证明教师过度自信会导致学生过拟合噪声
- 用教师预测熵值可选好老师,指导实际训练
对抗蒸馏旨在通过鲁棒教师的软标签提升学生模型的鲁棒性,但效果不稳定:更鲁棒的教师反而可能损害学生泛化能力。本文揭示其关键机制——教师监督信心与学生表征能力在一致不可学习数据子集(鲁棒不可学集)上的错位。我们构建两层神经网络的理论框架,证明当教师对不可学样本给出高置信度时,会迫使学生记忆虚假噪声模式,最终淹没鲁棒信号,导致鲁棒过拟合;反之,教师在这些样本上表现出高不确定性,则能抑制噪声记忆,使学生仅依赖可学习信号实现鲁棒泛化。我们在合成数据和真实图像分类数据集上验证该理论,确认鲁棒过拟合源于教师与不可学样本的交互。最后,证明教师在不可学样本上的预测熵是学生鲁棒性的强指标,为鲁棒教师选择提供理论指导。
原文摘要 · Abstract (English)
Adversarial Distillation aims to enhance student robustness by guiding the student with a robust teacher's soft labels within the min-max adversarial training framework, yet its success is notoriously inconsistent: a more robust teacher often fails to improve, or even harms, the student's robust generalization. In this paper, we identify a key mechanism of this teacher dependency: the misalignment between the teacher's supervisory confidence and the student's representational limitations on a consistent subset of training data -- the Robustly Unlearnable Set. We present a theoretical framework analyzing the feature learning dynamics of a two-layer neural network, demonstrating that this mismatch creates a dichotomy in distillation outcomes. We prove that when a teacher provides confident supervision on unlearnable samples, it compels the student to memorize spurious noise patterns that eventually overpower the learned robust signal, thereby driving robust overfitting. Conversely, a teacher that exhibits high uncertainty on these samples effectively suppresses noise memorization, allowing the student to rely solely on the learnable signal for robust generalization. We empirically validate our theory across both synthetic simulations and real-image classification datasets, confirming that robust overfitting is driven by the teacher's interaction with unlearnable samples. Finally, we demonstrate that a teacher's predictive entropy on unlearnable samples serves as a strong indicator of student robustness, validating our theoretical framework and offering a principled guideline for robust teacher selection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。