提出新方法解析知识蒸馏中的知识转移过程
On Explaining Knowledge Distillation: Measuring and Visualising the Knowledge Transfer Process
- 用梯度可视化方法UniCAM追踪教师到学生的知识传递路径
- 发现学生模型学会聚焦关键特征，忽略无关背景信息
- 引入两项新指标量化知识相关性，适用于模型调试与优化
知识蒸馏（KD）因教师到学生模型的知识转移过程不透明而面临挑战，难以解决相关问题。为此，本文提出UniCAM，一种基于梯度的可视化解释方法，可有效解析蒸馏过程中学习到的知识。实验表明，在教师知识指导下，学生模型更高效，学会提取相关特征并舍弃无关特征。我们将受教师指导学习的特征称为蒸馏特征，忽略的无关特征称为残余特征。蒸馏特征集中于输入的关键部分，如纹理和物体局部；而残余特征则呈现扩散注意力，常关注目标对象的背景等无关区域。此外，本文提出两个新指标：特征相似度分数（FSS）和相关性分数（RS），用于量化蒸馏知识的相关性。在CIFAR10、ASIRRA和Plant Disease数据集上的实验表明，UniCAM与两项指标为理解知识蒸馏过程提供了有价值洞察。
原文摘要 · Abstract (English)
Knowledge distillation (KD) remains challenging due to the opaque nature of the knowledge transfer process from a Teacher to a Student, making it difficult to address certain issues related to KD. To address this, we proposed UniCAM, a novel gradient-based visual explanation method, which effectively interprets the knowledge learned during KD. Our experimental results demonstrate that with the guidance of the Teacher's knowledge, the Student model becomes more efficient, learning more relevant features while discarding those that are not relevant. We refer to the features learned with the Teacher's guidance as distilled features and the features irrelevant to the task and ignored by the Student as residual features. Distilled features focus on key aspects of the input, such as textures and parts of objects. In contrast, residual features demonstrate more diffused attention, often targeting irrelevant areas, including the backgrounds of the target objects. In addition, we proposed two novel metrics: the feature similarity score (FSS) and the relevance score (RS), which quantify the relevance of the distilled knowledge. Experiments on the CIFAR10, ASIRRA, and Plant Disease datasets demonstrate that UniCAM and the two metrics offer valuable insights to explain the KD process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。