arXiv:2503.06307cs.CV2025-03ICCV被引 22

让学生模型动态学习老师,提升目标检测与分割精度。

ACAM-KD: Adaptive and Cooperative Attention Masking for Knowledge Distillation

  • 师生交互式特征融合,自适应调整注意力。
  • 动态生成空间与通道掩码,精准选择关键特征。
  • 适合追求高精度轻量化模型的开发者使用。

密集视觉预测任务(如检测与分割)对时序敏感应用(如自动驾驶、视频监控)至关重要。尽管深度模型表现优异,但其效率仍存挑战。知识蒸馏(KD)是有效的模型压缩技术,但现有基于特征的KD方法依赖静态、由教师主导的特征选择,无法适应学生模型的学习状态变化,也未能充分利用师生间的动态互动。为此,我们提出自适应师生协同注意力掩码知识蒸馏方法(ACAM-KD),包含两个核心组件:(1) 师生交叉注意力特征融合(STCA-FF),实现两模型特征的自适应融合,增强蒸馏交互性;(2) 自适应空间-通道掩码(ASCM),动态生成重要性掩码,提升空间与通道维度的特征选择能力。相比传统方法,ACAM-KD可随学生模型学习过程动态调整。在多个基准测试中验证了其有效性:在COCO2017上,将ResNet-50学生模型从ResNet-101教师模型蒸馏时,检测性能最高提升1.4 mAP;在Cityscapes语义分割任务中,使用DeepLabV3-MobileNetV2为学生模型时,mIoU较基线提升3.09。

原文摘要 · Abstract (English)

Dense visual prediction tasks, such as detection and segmentation, are crucial for time-critical applications (e.g., autonomous driving and video surveillance). While deep models achieve strong performance, their efficiency remains a challenge. Knowledge distillation (KD) is an effective model compression technique, but existing feature-based KD methods rely on static, teacher-driven feature selection, failing to adapt to the student's evolving learning state or leverage dynamic student-teacher interactions. To address these limitations, we propose Adaptive student-teacher Cooperative Attention Masking for Knowledge Distillation (ACAM-KD), which introduces two key components: (1) Student-Teacher Cross-Attention Feature Fusion (STCA-FF), which adaptively integrates features from both models for a more interactive distillation process, and (2) Adaptive Spatial-Channel Masking (ASCM), which dynamically generates importance masks to enhance both spatial and channel-wise feature selection. Unlike conventional KD methods, ACAM-KD adapts to the student's evolving needs throughout the entire distillation process. Extensive experiments on multiple benchmarks validate its effectiveness. For instance, on COCO2017, ACAM-KD improves object detection performance by up to 1.4 mAP over the state-of-the-art when distilling a ResNet-50 student from a ResNet-101 teacher. For semantic segmentation on Cityscapes, it boosts mIoU by 3.09 over the baseline with DeepLabV3-MobileNetV2 as the student model.

知识蒸馏目标检测图像分割注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。