arXiv:2603.03617cs.CV2026-03中稿 · CVPR被引 2

用语言描述提升红外可见光追踪的鲁棒性,解决外观变化和背景干扰问题。

RAGTrack: Language-aware RGBT Tracking with Retrieval-Augmented Generation

  • 引入多模态大模型自动生成文本描述,构建语言引导的追踪框架
  • 通过动态知识库与检索生成机制,实现跨时序语言推理与目标建模
  • 在四个基准上达到顶尖性能,适合复杂环境下的视觉追踪研究

RGB-Thermal(RGBT)追踪旨在通过融合可见光与热成像模态,在多样环境条件下实现稳健的目标定位。然而,现有方法仅依赖初始帧视觉信息建模目标,缺乏语言引导,无法适应外观变化。同时,现有方法存在搜索区域冗余和异构模态间隙问题,导致背景干扰。为此,我们首次在RGBT追踪基准中引入文本描述,通过多模态大模型(MLLMs)自动生成文本标注。随后提出RAGTrack,一种基于检索增强生成的新型框架。设计多模态变压器编码器(MTE)实现统一视觉-语言建模;提出自适应标记融合(ATF),依据跨模态相关性筛选目标相关标记并进行通道交换,缓解搜索冗余与模态差距;构建上下文感知推理模块(CRM),维护动态知识库,并利用检索增强生成(RAG)实现跨时序语言推理,增强目标建模能力。在四个RGBT基准上的大量实验表明,该框架在各类挑战场景下均达到最先进性能。源代码已公开:https://github.com/IdolLab/RAGTrack。

原文摘要 · Abstract (English)

RGB-Thermal (RGBT) tracking aims to achieve robust object localization across diverse environmental conditions by fusing visible and thermal infrared modalities. However, existing RGBT trackers rely solely on initial-frame visual information for target modeling, failing to adapt to appearance variations due to the absence of language guidance. Furthermore, current methods suffer from redundant search regions and heterogeneous modality gaps, causing background distraction. To address these issues, we first introduce textual descriptions into RGBT tracking benchmarks. This is accomplished through a pipeline that leverages Multi-modal Large Language Models (MLLMs) to automatically produce texual annotations. Afterwards, we propose RAGTrack, a novel Retrieval-Augmented Generation framework for robust RGBT tracking. To this end, we introduce a Multi-modal Transformer Encoder (MTE) for unified visual-language modeling. Then, we design an Adaptive Token Fusion (ATF) to select target-relevant tokens and perform channel exchanges based on cross-modal correlations, mitigating search redundancies and modality gaps. Finally, we propose a Context-aware Reasoning Module (CRM) to maintain a dynamic knowledge base and employ a Retrieval-Augmented Generation (RAG) to enable temporal linguistic reasoning for robust target modeling. Extensive experiments on four RGBT benchmarks demonstrate that our framework achieves state-of-the-art performance across various challenging scenarios. The source code is available https://github.com/IdolLab/RAGTrack.

RGBT追踪语言引导检索生成多模态建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。