拆解知识蒸馏中的信息损失，精准传递关键知识
CAKD: A Correlation-Aware Knowledge Distillation Framework Based on Decoupling Kullback-Leibler Divergence
- 将KL散度分解为三类相关性差异，识别影响预测的关键成分
- 在多个模型和数据集上，显著优于传统蒸馏方法
- 适合需要高效模型压缩与精度保持的场景
知识蒸馏中，通常关注如何转换和平衡多个蒸馏组件。本文强调需深入分析每个组件的重要性，发现并非所有部分都同等关键。基于此，我们将Kullback-Leibler（KL）散度解耦为三个独立成分：二分类差异（BCD）、强相关性差异（SCD）和弱相关性差异（WCD），它们对预测的影响程度不同。据此提出相关性感知的知识蒸馏框架（CAKD），优先保留对预测影响最大的蒸馏成分，以优化师生模型间知识迁移效率。实验表明，调节各成分权重能显著提升知识转化效果。进一步验证显示，该框架在多种模型和数据集上持续优于基线方法。本工作凸显了细致分析蒸馏过程各环节影响的重要性与有效性。
原文摘要 · Abstract (English)
In knowledge distillation, a primary focus has been on transforming and balancing multiple distillation components. In this work, we emphasize the importance of thoroughly examining each distillation component, as we observe that not all elements are equally crucial. From this perspective,we decouple the Kullback-Leibler (KL) divergence into three unique elements: Binary Classification Divergence (BCD), Strong Correlation Divergence (SCD), and Weak Correlation Divergence (WCD). Each of these elements presents varying degrees of influence. Leveraging these insights, we present the Correlation-Aware Knowledge Distillation (CAKD) framework. CAKD is designed to prioritize the facets of the distillation components that have the most substantial influence on predictions, thereby optimizing knowledge transfer from teacher to student models. Our experiments demonstrate that adjusting the effect of each element enhances the effectiveness of knowledge transformation. Furthermore, evidence shows that our novel CAKD framework consistently outperforms the baseline across diverse models and datasets. Our work further highlights the importance and effectiveness of closely examining the impact of different parts of distillation process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。