通过教师训练轨迹识别并消除模型捷径，提升学生模型泛化能力。
Anti-Shortcut Distillation via Temporal Negative Knowledge Transfer
- 用早期教师特征作为负样本，引导学生避开捷径方向。
- 在13组师生模型中，10组准确率超标准知识蒸馏，抗干扰能力更强。
- 适合追求模型鲁棒性与减少过拟合的开发者使用。
知识蒸馏通常让紧凑的学生模型向收敛的教师模型靠拢，却忽略了教师自身学习过程中被抑制的方向。本文观察到，教师优化轨迹中已编码了这些被抑制的捷径信息：早期教师强调但后期减弱的特征方向正是学生应避开的。为此提出反捷径蒸馏（ASD），将收敛教师 $ ext{T}_{\text{final}}$ 视为正向语义锚点，早期检查点教师 $ ext{T}_{\text{early}}$ 作为时间负参考。ASD结合两种损失：时间对比损失 $ ext{L}_{\text{tc}}$ 将早期教师特征设为同样本负例，在InfoNCE目标下与最终教师特征对比；以及捷径抑制损失 $ ext{L}_{\text{ss}}$，惩罚学生对 $ ext{E}[\mathbf{D}\mathbf{D}^{\top}]$ 最大特征向量的投影，该矩阵为早期至最终特征位移的未中心化二阶矩。在CIFAR-100、ImageNet-100和TinyImageNet上共13组师生对中，ASD在超过10组上取得最高干净准确率，且在12组上优于标准知识蒸馏。在CIFAR-100-C上的抗扰动测试中，其跨架构组合（WRN-40-2→ShuffleNet-V2）达到最低均值腐蚀误差（86.1 mCE）。机制诊断证实预期几何结构：ASD学生与捷径方向系统性反向对齐，而其在稳健子空间的投影显著更大（0.45 vs. 0.12）。
原文摘要 · Abstract (English)
Knowledge distillation (KD) trains a compact student by attracting it towards a converged teacher. It is silent about which directions the teacher itself learned to suppress: repulsive and bias-aware objectives exist, but none exploits the teacher's own trajectory to identify what the student should avoid. We observe that the missing signal is already encoded in the teacher's optimization trajectory: features that an early-stage teacher emphasizes but that a converged teacher attenuates are precisely the shortcut directions worth pushing the student away from. We instantiate this observation as \textbf{A}nti-\textbf{S}hortcut \textbf{D}istillation (ASD), a push--pull KD framework that treats the converged teacher $\Tfinal$ as a positive semantic anchor and an early-checkpoint teacher $\Tearly$ as a temporal negative reference. ASD couples two losses: a temporal contrastive loss ($\Ltc$) that places the early-teacher feature as a same-sample negative against in-batch and memory-bank final-teacher features in an InfoNCE objective; and a shortcut suppression loss ($\Lss$) that penalizes student projection onto the top eigenvectors of $\E[\Dh\Dh^{\top}]$, the uncentered second-moment matrix of early-to-final feature displacements. Across 13 teacher--student pairs on CIFAR-100, ImageNet-100, and TinyImageNet, ASD attains the highest clean top-1 accuracy on more than 10 pairs and outperforms standard KD on 12. On CIFAR-100-C corruption robustness, ASD obtains the lowest mean Corruption Error ($86.1$\,mCE) on the most challenging cross-architecture pair (WRN-40-2$\to$ShuffleNet-V2). Mechanistic diagnostics confirm the intended geometry: the ASD student is systematically anti-aligned with the shortcut direction, while its projection onto the robust subspace is substantially larger ($0.45$ vs.\ $0.12$).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。