用对比学习统一视觉语言特征,提升小目标追踪精度
COST: Contrastive One-Stage Transformer for Vision-Language Small Object Tracking
- 设计对比对齐机制,增强视觉与语言特征的语义一致性
- 仅用堆叠Transformer编码器即实现跨模态融合,性能超越现有方法
- 构建首个聚焦小目标的语言-视觉追踪数据集,支持高速与通用场景
Transformer在视觉-语言(VL)跟踪中展现出巨大潜力,但现有方法多依赖复杂多阶段多模态融合机制。直接融合忽略模态间特征分布差异,易导致表征不佳。本文提出COST,一种对比式单阶段Transformer融合框架,旨在学习语义一致且统一的VL表示。通过引入对比对齐策略,最大化视频与其对应语言描述间的互信息,实现有效的跨模态对齐,获得表示空间中的语义一致性特征。借助视觉-语言Transformer,建立高效多模态融合与推理机制,实证表明简单堆叠Transformer编码器即可实现统一的VL表示。此外,我们构建了新标注的VL追踪数据集VL-SOT500,包含两个挑战性子集:针对通用场景的VL-SOT230和针对高速场景的VL-SOT270,用于评估小目标追踪。小目标因外观弱、特征少而极难追踪,本数据集是首个探索利用语言线索增强小目标视觉表征的工作。大量实验表明,COST在五个现有VL追踪数据集及自建的VL-SOT500上均达到领先性能。源代码与数据集将公开。
原文摘要 · Abstract (English)
Transformer has recently demonstrated great potential in improving vision-language (VL) tracking algorithms. However, most of the existing VL trackers rely on carefully designed mechanisms to perform the multi-stage multi-modal fusion. Additionally, direct multi-modal fusion without alignment ignores distribution discrepancy between modalities in feature space, potentially leading to suboptimal representations. In this work, we propose COST, a contrastive one-stage transformer fusion framework for VL tracking, aiming to learn semantically consistent and unified VL representations. Specifically, we introduce a contrastive alignment strategy that maximizes mutual information (MI) between a video and its corresponding language description. This enables effective cross-modal alignment, yielding semantically consistent features in the representation space. By leveraging a visual-linguistic transformer, we establish an efficient multi-modal fusion and reasoning mechanism, empirically demonstrating that a simple stack of transformer encoders effectively enables unified VL representations. Moreover, we contribute a newly collected VL tracking benchmark dataset for small object tracking, named VL-SOT500, with bounding boxes and language descriptions. Our dataset comprises two challenging subsets, VL-SOT230 and VL-SOT270, dedicated to evaluating generic and high-speed small object tracking, respectively. Small object tracking is notoriously challenging due to weak appearance and limited features, and this dataset is, to the best of our knowledge, the first to explore the usage of language cues to enhance visual representation for small object tracking. Extensive experiments demonstrate that COST achieves state-of-the-art performance on five existing VL tracking datasets, as well as on our proposed VL-SOT500 dataset. Source codes and dataset will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。