arXiv:2507.20177cs.CVcs.MM2025-07TPAMI被引 60

一个模型搞定多种视频模态追踪,实时学习时间特征提升精度。

Towards Universal Modal Tracking with Online Dense Temporal Token Learning

  • 用视频级时序标记在线学习,捕捉目标运动轨迹和外观变化。
  • 单次训练适配多模态输入,在可见光与多模态数据上均达最新最好结果。
  • 适合需要跨模态统一追踪的工程应用,如自动驾驶与智能监控。

我们提出一种通用视频级模态感知追踪模型 { odtrack},支持RGB、RGB+热成像、RGB+深度、RGB+事件等多种模态任务,采用统一架构与参数。核心设计包括:(1)视频级采样,以序列方式建模全局上下文;(2)在线密集时序标记关联机制,通过视频流传播目标外观与运动信息;(3)可扩展模态支持,通过新型门控感知器与门控注意力自适应融合多模态特征,并以一次性训练压缩为统一参数用于多任务推理。该方法使净化后的标记序列可作为后续帧的时序提示,有效利用历史信息。相比需独立训练的多模态追踪器,本方案显著降低训练负担并增强表征能力。在可见光与多模态基准测试中,{ odtrack} 实现新最优性能。代码将开源于 https://github.com/GXNU-ZhongLab/ODTrack。

原文摘要 · Abstract (English)

We propose a universal video-level modality-awareness tracking model with online dense temporal token learning (called {\modaltracker}). It is designed to support various tracking tasks, including RGB, RGB+Thermal, RGB+Depth, and RGB+Event, utilizing the same model architecture and parameters. Specifically, our model is designed with three core goals: \textbf{Video-level Sampling}. We expand the model's inputs to a video sequence level, aiming to see a richer video context from an near-global perspective. \textbf{Video-level Association}. Furthermore, we introduce two simple yet effective online dense temporal token association mechanisms to propagate the appearance and motion trajectory information of target via a video stream manner. \textbf{Modality Scalable}. We propose two novel gated perceivers that adaptively learn cross-modal representations via a gated attention mechanism, and subsequently compress them into the same set of model parameters via a one-shot training manner for multi-task inference. This new solution brings the following benefits: (i) The purified token sequences can serve as temporal prompts for the inference in the next video frames, whereby previous information is leveraged to guide future inference. (ii) Unlike multi-modal trackers that require independent training, our one-shot training scheme not only alleviates the training burden, but also improves model representation. Extensive experiments on visible and multi-modal benchmarks show that our {\modaltracker} achieves a new \textit{SOTA} performance. The code will be available at https://github.com/GXNU-ZhongLab/ODTrack.

视频追踪多模态时序学习统一模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。