根据教师预测不确定性，选择性指导学生模型学习，提升压缩后图像分类精度。
Uncertainty-Aware Dual-Student Knowledge Distillation for Efficient Image Classification
- 用教师预测的不确定性筛选可信知识，指导两个不同架构的学生协同学习。
- 在ImageNet-100上，ResNet-18和MobileNetV2分别达83.84%和81.46%准确率。
- 适合追求高效且高精度模型压缩的研究者与工程应用。
知识蒸馏是模型压缩的重要技术，可将大模型知识迁移到小型学生模型。但传统方法对教师所有预测一视同仁，忽略其置信度。本文提出一种不确定性感知的双学生知识蒸馏框架，利用教师预测不确定性实现选择性引导。引入同伴学习机制，让两个异构学生模型（ResNet-18 和 MobileNetV2）同时从教师及彼此中学习。在 ImageNet-100 上的实验表明，该方法优于基线蒸馏方法：ResNet-18 达到 83.84% 的 top-1 准确率，较传统单学生方法提升 2.04%；MobileNetV2 达到 81.46%，提升 0.92%。
原文摘要 · Abstract (English)
Knowledge distillation has emerged as a powerful technique for model compression, enabling the transfer of knowledge from large teacher networks to compact student models. However, traditional knowledge distillation methods treat all teacher predictions equally, regardless of the teacher's confidence in those predictions. This paper proposes an uncertainty-aware dual-student knowledge distillation framework that leverages teacher prediction uncertainty to selectively guide student learning. We introduce a peer-learning mechanism where two heterogeneous student architectures, specifically ResNet-18 and MobileNetV2, learn collaboratively from both the teacher network and each other. Experimental results on ImageNet-100 demonstrate that our approach achieves superior performance compared to baseline knowledge distillation methods, with ResNet-18 achieving 83.84\% top-1 accuracy and MobileNetV2 achieving 81.46\% top-1 accuracy, representing improvements of 2.04\% and 0.92\% respectively over traditional single-student distillation approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。