提升长视频中任意点的鲁棒追踪能力,通过时空上下文增强特征查询。
TAPTRv3: Spatial and Temporal Context Foster Robust Tracking of Any Point in Long Video
- 引入时空上下文注意力机制,改进点级特征查询质量。
- 在多个长视频数据集上超越TAPTRv2,达当前最优性能。
- 适合需要高精度长时序追踪的应用场景,如医疗视频分析。
本文在TAPTRv2基础上提出TAPTRv3。TAPTRv2是基于DETR思想的点追踪框架,在常规视频中表现良好,但在长视频中因目标点随时间变化加剧而失效。TAPTRv3通过引入空间与时间上下文,提升长视频中空间和时间维度上的特征查询鲁棒性。针对空间特征查询,提出上下文感知交叉注意力(CCA),将空间上下文融入注意力机制以优化图像特征查询;针对时间特征查询,设计可见性感知长时序注意力(VLTA),在关注历史帧的同时考虑其可见性,有效缓解了TAPTRv2中由类似RNN的长期建模导致的特征漂移问题。TAPTRv3在多数挑战性数据集上大幅超越TAPTRv2,并达到当前最优水平,即使对比使用大规模内部数据训练的方法也保持优势。
原文摘要 · Abstract (English)
In this paper, built upon TAPTRv2, we present TAPTRv3. TAPTRv2 is a simple yet effective DETR-like point tracking framework that works fine in regular videos but tends to fail in long videos. TAPTRv3 improves TAPTRv2 by addressing its shortcomings in querying high-quality features from long videos, where the target tracking points normally undergo increasing variation over time. In TAPTRv3, we propose to utilize both spatial and temporal context to bring better feature querying along the spatial and temporal dimensions for more robust tracking in long videos. For better spatial feature querying, we identify that off-the-shelf attention mechanisms struggle with point-level tasks and present Context-aware Cross-Attention (CCA). CCA introduces spatial context into the attention mechanism to enhance the quality of attention scores when querying image features. For better temporal feature querying, we introduce Visibility-aware Long-Temporal Attention (VLTA), which conducts temporal attention over past frames while considering their corresponding visibilities. This effectively addresses the feature drifting problem in TAPTRv2 caused by its RNN-like long-term modeling. TAPTRv3 surpasses TAPTRv2 by a large margin on most of the challenging datasets and obtains state-of-the-art performance. Even when compared with methods trained on large-scale extra internal data, TAPTRv3 still demonstrates superiority.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。