揭秘知识蒸馏中学生模型学到了什么，提出无需教师的自蒸馏新方法。
What Do Students Learn? A Feature-Level Analysis of Dark Knowledge
- 用交互张量分析学生模型特征学习机制
- 发现蒸馏会抑制低频、样本特有特征，保留通用特征
- 基于混淆矩阵自生成软标签，提升性能且更高效
知识蒸馏（KD）是模型压缩的强大工具，但学生模型如何习得特征表示仍不清晰。本文采用交互张量框架分析学生特征学习过程，发现有效蒸馏起到正则化作用，剔除低频、样本特异的特征，促使学生依赖一组高复用性特征。关键发现：数据集级混淆矩阵包含与教师‘暗知识’类似的结构信息。据此提出混淆蒸馏（CD），一种无需教师的自蒸馏方法，利用模型自身演化的混淆模式作为动态软目标。在CIFAR-100上，CD在ResNet-34和ResNet-50上表现优异，较CS-KD和PS-KD提升1.2%，且计算效率优于标准KD。
原文摘要 · Abstract (English)
Knowledge Distillation (KD) is a powerful tool for model compression, yet the precise mechanisms by which student models acquire feature representations remain underexplored. In this work, we analyze student feature learning using the Interaction Tensor framework. Our analysis reveals that effective KD acts as a regularizer that prunes low-frequency, sample-specific features, encouraging the student to rely on a compact set of highly reusable features. Crucially, we observe that the dataset-level confusion matrix contains structural information analogous to the teacher's "Dark Knowledge." Leveraging this insight, we propose Confusion Distillation (CD), a teacher-free self-distillation method that utilizes the model's own evolving )confusion patterns as dynamic soft targets. CD achieves competitive performance on ResNet-34 and ResNet-50 for CIFAR-100, outperforming existing self-distillation methods like CS-KD and PS-KD by 1.2% while offering a computationally efficient alternative to standard KD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。