用扩散模型让学生自学习教师知识，提升模型压缩效果
Teacher-Guided Student Self-Knowledge Distillation Using Diffusion Model
- 用轻量扩散模型引导学生特征去噪，模拟教师知识
- 通过局部敏感哈希对比原始与去噪特征，提取有效知识
- 在多个数据集上超越现有方法，适合模型压缩场景
现有知识蒸馏方法通常通过有意义的特征处理和损失函数实现教师与学生之间的特征对齐。然而，由于教师与学生特征分布存在差异，学生可能学到不兼容的信息。为解决此问题，我们提出教师引导的学生扩散自知识蒸馏方法（DSKD）。不同于直接对齐教师与学生特征，我们利用教师分类器指导轻量级扩散模型的去噪采样过程，使学生特征在去噪后蕴含教师知识，可视为教师角色。进而提出一种新型局部敏感哈希（LSH）引导的特征蒸馏方法，比较原始与去噪后的学生特征。该方法有效消除教师与学生间映射方式和特征分布的差异，同时学习到有意义的知识。在视觉识别任务上的实验表明，DSKD在多种模型与数据集上显著优于现有方法。代码附于补充材料。
原文摘要 · Abstract (English)
Existing Knowledge Distillation (KD) methods often align feature information between teacher and student by exploring meaningful feature processing and loss functions. However, due to the difference in feature distributions between the teacher and student, the student model may learn incompatible information from the teacher. To address this problem, we propose teacher-guided student Diffusion Self-KD, dubbed as DSKD. Instead of the direct teacher-student alignment, we leverage the teacher classifier to guide the sampling process of denoising student features through a light-weight diffusion model. We then propose a novel locality-sensitive hashing (LSH)-guided feature distillation method between the original and denoised student features. The denoised student features encapsulate teacher knowledge and could be regarded as a teacher role. In this way, our DSKD method could eliminate discrepancies in mapping manners and feature distributions between the teacher and student, while learning meaningful knowledge from the teacher. Experiments on visual recognition tasks demonstrate that DSKD significantly outperforms existing KD methods across various models and datasets. Our code is attached in supplementary material.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。