提出STTrack,用时空联合建模提升多模态视频目标追踪精度
Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking
- 设计时序状态生成器,持续生成含时序信息的令牌
- 在5个数据集上达到当前最优性能,显著提升复杂场景追踪效果
- 适合需要高精度多模态追踪的应用场景
多模态追踪因能有效克服传统RGB追踪的固有局限而受到广泛关注。然而,现有方法主要聚焦于空间特征融合或仅利用帧间稀疏时序关系,未能充分挖掘多模态视频中的时序相关性,难以捕捉目标在复杂场景下的动态变化与运动信息。为此,我们提出统一的多模态时空追踪框架STTrack。不同于以往仅依赖参考信息更新的范式,我们引入时序状态生成器(TSG),持续生成包含多模态时序信息的令牌序列,用于指导下一时刻的目标定位、建立长程帧间上下文关联并捕捉目标时序轨迹。在空间层面,引入Mamba融合与背景抑制交互(BSI)模块,构建双阶段跨模态信息交互与融合机制。在五个基准数据集上的大量对比实验表明,STTrack在多种多模态追踪场景中均达到先进水平。代码已公开:https://github.com/NJU-PCALab/STTrack。
原文摘要 · Abstract (English)
Multimodal tracking has garnered widespread attention as a result of its ability to effectively address the inherent limitations of traditional RGB tracking. However, existing multimodal trackers mainly focus on the fusion and enhancement of spatial features or merely leverage the sparse temporal relationships between video frames. These approaches do not fully exploit the temporal correlations in multimodal videos, making it difficult to capture the dynamic changes and motion information of targets in complex scenarios. To alleviate this problem, we propose a unified multimodal spatial-temporal tracking approach named STTrack. In contrast to previous paradigms that solely relied on updating reference information, we introduced a temporal state generator (TSG) that continuously generates a sequence of tokens containing multimodal temporal information. These temporal information tokens are used to guide the localization of the target in the next time state, establish long-range contextual relationships between video frames, and capture the temporal trajectory of the target. Furthermore, at the spatial level, we introduced the mamba fusion and background suppression interactive (BSI) modules. These modules establish a dual-stage mechanism for coordinating information interaction and fusion between modalities. Extensive comparisons on five benchmark datasets illustrate that STTrack achieves state-of-the-art performance across various multimodal tracking scenarios. Code is available at: https://github.com/NJU-PCALab/STTrack.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。