动态更新语言描述,让视觉语言追踪更准更稳
Dynamic Updates for Language Adaptation in Visual-Language Tracking
- 用大模型根据视觉特征实时生成目标语言描述
- 在多个基准上达到新最佳性能,最高提升4.2%
- 适合需要精准追踪动态目标的场景应用
视觉语言追踪中,多模态参考与追踪目标之间的语义一致性至关重要。现有框架依赖静态多模态参考定位动态目标,易产生语义偏差,降低追踪鲁棒性。为此,我们提出新框架DUTrack,通过动态更新多模态参考以保持一致性。具体而言,引入动态语言更新模块,利用大语言模型基于视觉特征和类别信息生成动态语言描述;设计动态模板捕获模块,提取与描述高度匹配的图像区域;并制定基于目标位移、尺度等变化的更新策略以保证效率。最终,将最新状态的动态模板与语言描述用于更新多模态参考,提升后续推理精度。DUTrack在四个主流视觉语言及两个纯视觉追踪基准(LaSOT、LaSOT$_{\rm{ext}}$、TNL2K、OTB99-Lang、GOT-10K、UAV123)上均取得新最优结果。代码与模型已开源。
原文摘要 · Abstract (English)
The consistency between the semantic information provided by the multi-modal reference and the tracked object is crucial for visual-language (VL) tracking. However, existing VL tracking frameworks rely on static multi-modal references to locate dynamic objects, which can lead to semantic discrepancies and reduce the robustness of the tracker. To address this issue, we propose a novel vision-language tracking framework, named DUTrack, which captures the latest state of the target by dynamically updating multi-modal references to maintain consistency. Specifically, we introduce a Dynamic Language Update Module, which leverages a large language model to generate dynamic language descriptions for the object based on visual features and object category information. Then, we design a Dynamic Template Capture Module, which captures the regions in the image that highly match the dynamic language descriptions. Furthermore, to ensure the efficiency of description generation, we design an update strategy that assesses changes in target displacement, scale, and other factors to decide on updates. Finally, the dynamic template and language descriptions that record the latest state of the target are used to update the multi-modal references, providing more accurate reference information for subsequent inference and enhancing the robustness of the tracker. DUTrack achieves new state-of-the-art performance on four mainstream vision-language and two vision-only tracking benchmarks, including LaSOT, LaSOT$_{\rm{ext}}$, TNL2K, OTB99-Lang, GOT-10K, and UAV123. Code and models are available at https://github.com/GXNU-ZhongLab/DUTrack.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。