提出三路切换框架,提升跨模态目标追踪精度与稳定性
SwiTrack: Tri-State Switch for Cross-Modal Object Tracking
- 设计三路结构:视觉编码器处理可见光,NIR适配器优化红外,轨迹预测补全无效模态
- 在最新基准上精度和成功率分别提升7.2%和4.3%,达实时65帧/秒
- 适合需要跨模态鲁棒追踪的智能监控、自动驾驶场景
跨模态目标追踪(CMOT)是一项新兴任务,旨在视频流在不同模态间切换时保持目标一致性,每帧仅提供单一模态数据,主要关注可见光-近红外(RGB-NIR)追踪。现有方法通常将并行的RGB与NIR分支连接至共享主干网络,限制了模态特异性特征的充分提取,且难以应对不可靠输入导致的目标漂移问题。本文提出SwiTrack,一种新型状态切换框架,通过部署三个专用流重新定义CMOT。具体而言,可见光帧由视觉编码器处理,近红外帧则通过与视觉编码器耦合的NIR门控适配器进行逐步校准,以增强共享潜在空间特征;对于无效模态,一致性轨迹预测模块利用时空线索估计目标运动,确保追踪鲁棒性并缓解漂移。此外,引入动态模板重建机制迭代更新模板特征,并采用相似性对齐损失强化特征一致性。在最新基准上的实验结果表明,该追踪器达到当前最优性能,精度率和成功率分别提升7.2%和4.3%,同时保持65帧/秒的实时追踪速度。代码与模型已在https://github.com/xuboyue1999/SwiTrack.git发布。
原文摘要 · Abstract (English)
Cross-modal object tracking (CMOT) is an emerging task that maintains target consistency while the video stream switches between different modalities, with only one modality available in each frame, mostly focusing on RGB-Near Infrared (RGB-NIR) tracking. Existing methods typically connect parallel RGB and NIR branches to a shared backbone, which limits the comprehensive extraction of distinctive modality-specific features and fails to address the issue of object drift, especially in the presence of unreliable inputs. In this paper, we propose SwiTrack, a novel state-switching framework that redefines CMOT through the deployment of three specialized streams. Specifically, RGB frames are processed by the visual encoder, while NIR frames undergo refinement via a NIR gated adapter coupled with the visual encoder to progressively calibrate shared latent space features, thereby yielding more robust cross-modal representations. For invalid modalities, a consistency trajectory prediction module leverages spatio-temporal cues to estimate target movement, ensuring robust tracking and mitigating drift. Additionally, we incorporate dynamic template reconstruction to iteratively update template features and employ a similarity alignment loss to reinforce feature consistency. Experimental results on the latest benchmarks demonstrate that our tracker achieves state-of-the-art performance, boosting precision rate and success rate gains by 7.2\% and 4.3\%, respectively, while maintaining real-time tracking at 65 frames per second. Code and models are available at https://github.com/xuboyue1999/SwiTrack.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。