arXiv:2505.15133cs.CVcs.AI2025-05NeurIPS被引 1

通过解耦与去噪提升知识蒸馏效果,解决混淆信息干扰问题。

DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer

  • 分层解耦任务与非任务梯度,用独立动量更新避免信号干扰。
  • 动态筛选高置信度非目标类知识,早期净化暗知识降低噪声。
  • 在多个数据集上显著超越现有方法,适合追求高精度蒸馏的研究者。

近期知识蒸馏研究强调解耦不同知识成分的重要性。现有方法虽利用动量机制分离任务导向与蒸馏梯度,却忽略了目标类与非目标类知识流之间的内在冲突。此外,非目标类中的低置信度暗知识引入噪声信号,阻碍有效知识迁移。为此,我们提出 DeepKD,一种融合双层解耦与自适应去噪的新训练框架。首先,基于任务导向与非任务导向知识蒸馏中梯度信噪比(GSNR)特性的理论分析,为各成分设计独立动量更新器,防止相互干扰。我们发现,任务导向梯度(TOG)、目标类梯度(TCG)与非目标类梯度(NCG)的最优动量系数应与其GSNR正相关。其次,引入动态Top-k掩码(DTM)机制,训练初期从较小的K值开始,逐步增加以纳入更多非目标类,遵循课程学习原则。DTM联合过滤教师与学生模型的低置信度输出,有效净化早期训练中的暗知识。在CIFAR-100、ImageNet和MS-COCO上的大量实验验证了DeepKD的有效性。代码已公开于https://github.com/haiduo/DeepKD。

原文摘要 · Abstract (English)

Recent advances in knowledge distillation have emphasized the importance of decoupling different knowledge components. While existing methods utilize momentum mechanisms to separate task-oriented and distillation gradients, they overlook the inherent conflict between target-class and non-target-class knowledge flows. Furthermore, low-confidence dark knowledge in non-target classes introduces noisy signals that hinder effective knowledge transfer. To address these limitations, we propose DeepKD, a novel training framework that integrates dual-level decoupling with adaptive denoising. First, through theoretical analysis of gradient signal-to-noise ratio (GSNR) characteristics in task-oriented and non-task-oriented knowledge distillation, we design independent momentum updaters for each component to prevent mutual interference. We observe that the optimal momentum coefficients for task-oriented gradient (TOG), target-class gradient (TCG), and non-target-class gradient (NCG) should be positively related to their GSNR. Second, we introduce a dynamic top-k mask (DTM) mechanism that gradually increases K from a small initial value to incorporate more non-target classes as training progresses, following curriculum learning principles. The DTM jointly filters low-confidence logits from both teacher and student models, effectively purifying dark knowledge during early training. Extensive experiments on CIFAR-100, ImageNet, and MS-COCO demonstrate DeepKD's effectiveness. Our code is available at https://github.com/haiduo/DeepKD.

知识蒸馏深度学习模型压缩去噪

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。