让追踪模型能自动识别并跟踪任意新物体,解决真实场景中物体多样性的难题。
COVTrack++: Learning Open-Vocabulary Multi-Object Tracking from Continuous Videos via a Synergistic Paradigm
- 构建连续标注数据集C-TAO,提升标注密度26倍,捕捉物体运动细节。
- 提出协同框架COVTrack++,在检测与关联间实现双向优化,提升追踪稳定性。
- 首次实现零样本泛化,适用于复杂动态场景中的多目标追踪任务。
多目标追踪(MOT)传统上仅针对有限类别,限制了其在真实场景中对多样化物体的应用。开放词汇多目标追踪(OVMOT)通过支持任意类别(包括训练中未见的新物体)来克服这一局限。然而,当前进展受限于两大挑战:缺乏连续标注的视频训练数据,以及缺少可协同处理检测与关联的定制化框架。为此,我们构建了首个用于OVMOT的连续标注数据集C-TAO,其标注密度相较原始TAO提升26倍,有效捕捉平滑运动动态与中间物体状态。针对框架瓶颈,提出COVTrack++协同框架,通过三个模块实现检测与关联的双向互促:(1) 多线索自适应融合(MCF)动态平衡外观、运动与语义线索以优化关联特征;(2) 多粒度层次聚合(MGA)利用密集检测中的层级空间关系,通过可见子节点(如物体部件)辅助遮挡父对象(如完整躯干)的特征增强;(3) 时间置信度传播(TCP)通过高置信度轨迹提升低置信度候选帧,缓解闪烁问题,稳定轨迹。在TAO上的大量实验表明,该方法达到领先性能,新指标TETA在验证集和测试集分别达35.4%和30.5%,相比之前方法提升4.8%(novel AssocA)和5.8%(novel LocA),并在BDD100K上展现强劲零样本泛化能力。
原文摘要 · Abstract (English)
Multi-Object Tracking (MOT) has traditionally focused on a few specific categories, restricting its applicability to real-world scenarios involving diverse objects. Open-Vocabulary Multi-Object Tracking (OVMOT) addresses this by enabling tracking of arbitrary categories, including novel objects unseen during training. However, current progress is constrained by two challenges: the lack of continuously annotated video data for training, and the lack of a customized OVMOT framework to synergistically handle detection and association. We address the data bottleneck by constructing C-TAO, the first continuously annotated training set for OVMOT, which increases annotation density by 26x over the original TAO and captures smooth motion dynamics and intermediate object states. For the framework bottleneck, we propose COVTrack++, a synergistic framework that achieves a bidirectional reciprocal mechanism between detection and association through three modules: (1) Multi-Cue Adaptive Fusion (MCF) dynamically balances appearance, motion, and semantic cues for association feature learning; (2) Multi-Granularity Hierarchical Aggregation (MGA) exploits hierarchical spatial relationships in dense detections, where visible child nodes (e.g., object parts) assist occluded parent objects (e.g., whole body) for association feature enhancement; (3) Temporal Confidence Propagation (TCP) recovers flickering detections through high-confidence tracked objects boosting low-confidence candidates across frames, stabilizing trajectories. Extensive experiments on TAO demonstrate state-of-the-art performance, with novel TETA reaching 35.4% and 30.5% on validation and test sets, improving novel AssocA by 4.8% and novel LocA by 5.8% over previous methods, and show strong zero-shot generalization on BDD100K.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。