用多维度语言描述提升视觉追踪鲁棒性，解决目标外观变化难题。
CLDTracker: A Comprehensive Language Description for Visual Tracking
- 构建文本与视觉双分支架构，融合CLIP和GPT-4V生成丰富语义描述
- 在6个标准数据集上达到最新最优性能（SOTA），跟踪准确率显著提升
- 适合需要高精度、强泛化能力的视觉追踪应用场景
视觉目标追踪（VOT）因目标外观动态变化、遮挡和背景干扰而始终是计算机视觉中的核心挑战。传统追踪器主要依赖视觉线索，在复杂场景下表现不佳。近年来，视觉语言模型（VLMs）在开放词汇检测和图像描述等任务中展现出语义理解潜力，提示其在VOT中的应用前景。然而，直接应用VLMs于VOT面临三大瓶颈：缺乏能充分刻画目标细微语义的丰富文本表征；视觉与文本特征融合机制低效，难以实现整体理解；且缺乏对目标随时间演化的外观在语言层面的建模，导致初始描述与后续视觉变化脱节。为弥合这些差距并释放VLM在追踪中的潜力，我们提出CLDTracker——一种全面的语言描述框架用于鲁棒视觉追踪。该框架采用双分支结构，文本分支利用CLIP和GPT-4V生成富含语义与上下文信息的多重文本描述，解决文本表征贫乏问题；视觉分支则同步处理图像特征。在六个标准VOT基准上的实验表明，CLDTracker实现当前最优性能，验证了鲁棒且时序自适应的视觉语言表征在追踪中的有效性。代码与模型已公开于：https://github.com/HamadYA/CLDTracker
原文摘要 · Abstract (English)
VOT remains a fundamental yet challenging task in computer vision due to dynamic appearance changes, occlusions, and background clutter. Traditional trackers, relying primarily on visual cues, often struggle in such complex scenarios. Recent advancements in VLMs have shown promise in semantic understanding for tasks like open-vocabulary detection and image captioning, suggesting their potential for VOT. However, the direct application of VLMs to VOT is hindered by critical limitations: the absence of a rich and comprehensive textual representation that semantically captures the target object's nuances, limiting the effective use of language information; inefficient fusion mechanisms that fail to optimally integrate visual and textual features, preventing a holistic understanding of the target; and a lack of temporal modeling of the target's evolving appearance in the language domain, leading to a disconnect between the initial description and the object's subsequent visual changes. To bridge these gaps and unlock the full potential of VLMs for VOT, we propose CLDTracker, a novel Comprehensive Language Description framework for robust visual Tracking. Our tracker introduces a dual-branch architecture consisting of a textual and a visual branch. In the textual branch, we construct a rich bag of textual descriptions derived by harnessing the powerful VLMs such as CLIP and GPT-4V, enriched with semantic and contextual cues to address the lack of rich textual representation. Experiments on six standard VOT benchmarks demonstrate that CLDTracker achieves SOTA performance, validating the effectiveness of leveraging robust and temporally-adaptive vision-language representations for tracking. Code and models are publicly available at: https://github.com/HamadYA/CLDTracker
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。