不用分类分数损失，纯用特征损失实现更强的模型压缩。
Logit-Based Losses Limit the Effectiveness of Feature Knowledge Distillation
- 仅使用中间层特征损失训练学生模型，摒弃传统分类分数损失。
- 在三个数据集上提升最高达15%的准确率，优于现有方法。
- 适用于各类卷积和视觉变压器结构，适合追求高性能压缩的研究者。
知识蒸馏（KD）方法可将参数量大的教师模型知识迁移至轻量化学生模型。当前特征知识蒸馏方法通常结合基于logits（即softmax前的类别得分）的损失与中间层特征损失。不同于以往方法，本文提出一种仅使用特征损失（不依赖交叉熵等logit损失）训练学生主干网络的特征蒸馏框架。借助近期关于潜在表示几何结构的发现，我们引入一个知识质量度量，用于识别哪些教师层提供最有效的知识。在三个图像分类数据集上，采用四种不同学生-教师组合（涵盖卷积神经网络与视觉变换器）的实验表明，本方法达到业界领先性能，相比标准方法最高提升15%的准确率。代码已公开，便于后续研究：https://github.com/Thegolfingocto/KD_wo_CE。
原文摘要 · Abstract (English)
Knowledge distillation (KD) methods can transfer knowledge of a parameter-heavy teacher model to a light-weight student model. The status quo for feature KD methods is to utilize loss functions based on logits (i.e., pre-softmax class scores) and intermediate layer features (i.e., latent representations). Unlike previous approaches, we propose a feature KD framework for training the student's backbone using feature-based losses exclusively (i.e., without logit-based losses such as cross entropy). Leveraging recent discoveries about the geometry of latent representations, we introduce a knowledge quality metric for identifying which teacher layers provide the most effective knowledge for distillation. Experiments on three image classification datasets with four diverse student-teacher pairs, spanning convolutional neural networks and vision transformers, demonstrate our KD method achieves state-of-the-art performance, delivering top-1 accuracy boosts of up to 15% over standard approaches. We publically share our code to facilitate future work at https://github.com/Thegolfingocto/KD_wo_CE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。