用跨注意力机制让学生模型更全面学习教师模型的像素关系
CanKD: Cross-Attention-based Non-local operation for Feature-based Knowledge Distillation
- 用跨注意力让学生每像素关注教师所有像素，实现非局部特征对齐
- 在目标检测和分割任务上超越现有方法，提升显著且仅加一个损失项
- 适合需要高精度特征迁移的视觉任务，如小模型压缩与部署
我们提出一种基于交叉注意力的非局部知识蒸馏框架（CanKD），通过引入跨注意力机制增强特征级知识迁移。与传统自注意力方法独立对齐师生特征图不同，CanKD使学生特征图中的每个像素能动态关注教师特征图的所有像素，从而更充分地捕捉像素间关系，提升特征表示能力。该方法仅增加一个额外损失函数，便在目标检测与图像分割任务中优于现有注意力引导的蒸馏方法。大量实验表明，CanKD在特征蒸馏与混合蒸馏方法上均达到领先性能，展现出其作为计算机视觉中注意力引导蒸馏新范式的重要潜力。代码已开源：https://github.com/tori-hotaru/CanKD。
原文摘要 · Abstract (English)
We propose Cross-Attention-based Non-local Knowledge Distillation (CanKD), a novel feature-based knowledge distillation framework that leverages cross-attention mechanisms to enhance the knowledge transfer process. Unlike traditional self-attention-based distillation methods that align teacher and student feature maps independently, CanKD enables each pixel in the student feature map to dynamically consider all pixels in the teacher feature map. This non-local knowledge transfer more thoroughly captures pixel-wise relationships, improving feature representation learning. Our method introduces only an additional loss function to achieve superior performance compared with existing attention-guided distillation methods. Extensive experiments on object detection and image segmentation tasks demonstrate that CanKD outperforms state-of-the-art feature and hybrid distillation methods. These experimental results highlight CanKD's potential as a new paradigm for attention-guided distillation in computer vision tasks. Code is available at https://github.com/tori-hotaru/CanKD
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。