arXiv:2410.11586cs.CV2024-10中稿 · ACM MM2024被引 31

通过联合知识蒸馏破解可见光与红外图像模态差异,提升跟踪性能。

Breaking Modality Gap in RGBT Tracking: Coupled Knowledge Distillation

  • 设计双学生网络,用风格蒸馏缩小跨模态图像风格差距。
  • 在保持内容信息的同时,实现96.4帧/秒的高速跟踪。
  • 适合需要高精度、实时跨模态跟踪的应用场景。

可见光(RGB)与热红外(TIR)图像之间的模态差异是现有RGBT跟踪方法中常被忽视的关键问题,其主要表现为图像风格差异。本文提出一种新型耦合知识蒸馏框架CKD,旨在通过追求不同模态的共性风格来打破模态间隙,实现高性能的RGBT跟踪。具体地,引入两个学生网络,并采用风格蒸馏损失使它们的风格特征尽可能一致。为避免风格蒸馏损害内容表征,以原始的RGB和TIR网络作为教师,通过风格-内容正交特征解耦机制,分别将内容知识蒸馏至两个学生网络。上述两种蒸馏过程在在线优化框架中耦合,生成无模态间隙的跨模态新特征表示。此外,还设计了掩码建模策略和多模态候选令牌剔除策略,分别提升跟踪鲁棒性与效率。在五个标准RGBT跟踪数据集上的大量实验表明,该方法优于当前最优方法,且达到96.4帧/秒的最快跟踪速度。代码已开源。

原文摘要 · Abstract (English)

Modality gap between RGB and thermal infrared (TIR) images is a crucial issue but often overlooked in existing RGBT tracking methods. It can be observed that modality gap mainly lies in the image style difference. In this work, we propose a novel Coupled Knowledge Distillation framework called CKD, which pursues common styles of different modalities to break modality gap, for high performance RGBT tracking. In particular, we introduce two student networks and employ the style distillation loss to make their style features consistent as much as possible. Through alleviating the style difference of two student networks, we can break modality gap of different modalities well. However, the distillation of style features might harm to the content representations of two modalities in student networks. To handle this issue, we take original RGB and TIR networks as the teachers, and distill their content knowledge into two student networks respectively by the style-content orthogonal feature decoupling scheme. We couple the above two distillation processes in an online optimization framework to form new feature representations of RGB and thermal modalities without modality gap. In addition, we design a masked modeling strategy and a multi-modal candidate token elimination strategy into CKD to improve tracking robustness and efficiency respectively. Extensive experiments on five standard RGBT tracking datasets validate the effectiveness of the proposed method against state-of-the-art methods while achieving the fastest tracking speed of 96.4 FPS. Code available at https://github.com/Multi-Modality-Tracking/CKD.

RGBT跟踪知识蒸馏跨模态实时跟踪

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。