任意模态统一追踪框架,跨模态自适应更灵活。
AnyTrack: Unifying Visual Object Tracking with Any Modalities

- 设计动态交互模块,自动适配不同模态输入
- 在多模态缺失下仍保持精准定位,性能领先
- 支持视觉、语言、音频等任意组合,适合多场景应用
视觉目标追踪旨在连续帧中定位特定目标,已从单模态发展到多模态。然而,现有方法通常针对固定模态组合设计,需为不同输入分别建模,导致对缺失或低质模态适应性差,泛化能力有限。为此,我们提出统一框架AnyTrack,实现任意模态下的目标追踪。设计模态感知交互模块(MIM),动态融合异构模态,弥合模态差异并聚合时序信息以维持时空一致性;引入上下文理解模块(CUM),通过全局-局部提示建立视觉特征与目标位置的空间对应关系,实现目标感知的上下文建模,提升前景-背景区分度。同时扩展现有多模态追踪基准,加入灰度图、语言描述和音频片段。在完整与缺失模态设置下大量实验表明,AnyTrack达到最先进性能,验证其有效性与灵活性。代码开源:https://github.com/IdolLab/AnyTrack。
原文摘要 · Abstract (English)
Visual object tracking aims to continuously locate specific targets within sequential frames, evolving from single-modal methods to multi-modal ones. However, existing multi-modal trackers are typically designed for fixed modality combinations, requiring separate models for different inputs. This leads to a poor adaptability to missing or imperfect modalities, and limited generalization. To address these issues, we propose a novel unified framework called AnyTrack for object tracking with any modalities. Specifically, we design a Modality-aware Interaction Module (MIM) to facilitate dynamic interaction across diverse modalities. This module bridges modality discrepancies and aggregates temporal cues to maintain spatio-temporal consistency during cross-modal interaction. Furthermore, we introduce a Context Understanding Module (CUM) to establish spatial correspondence between visual features and target locations via global-local prompts. This module employs target-aware context modeling to enhance foreground-background discrimination for precise localization. Finally, to support the training and evaluation under diverse modalities, we extend existing multi-modal object tracking benchmarks by incorporating grayscale images, language descriptions, and audio clips. Extensive experiments with both complete and missing modality settings demonstrate that our AnyTrack achieves state-of-the-art performance, validating its effectiveness and flexibility. The source code is available at https://github.com/IdolLab/AnyTrack.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。