提出新追踪任务ReaMOT,让模型通过逻辑推理跟踪语言指令指定的目标。
ReaMOT: A Benchmark and Framework for Reasoning-based Multi-Object Tracking
- 将追踪升级为需逻辑推理的认知任务,突破传统视觉-文本匹配局限。
- 构建包含1156条指令、42万图像-语言对的大规模数据集,75%以上为高阶推理任务。
- 提出ReaTrack框架,解耦语义推理与运动连续性,显著提升复杂指令追踪效果。
指代式多目标追踪(RMOT)旨在根据语言指令追踪目标,但现有方法严重依赖显式视觉-文本匹配,难以泛化到需逻辑推理的复杂指令。为此,我们提出基于推理的多目标追踪(ReaMOT),将追踪提升至认知层面,要求模型通过逻辑推理推断并追踪满足隐式约束的目标。为推动该领域发展,我们构建了ReaMOT挑战赛,包含定制度量体系和大规模数据集。该数据集涵盖1,156条语言指令、423,359个图像-语言对和869个视频序列,系统分为六个评估场景,其中超过75%的指令专注于高阶推理。针对传统追踪器缺乏认知能力且直接使用大视觉语言模型(LVLM)导致严重时间不一致性的问题,我们提出ReaTrack。该训练免于框架基于‘解耦高层语义定位与低层运动连续性’的洞察,动态对齐思维型LVLM的语义检测与SAM2的鲁棒运动先验。在ReaMOT挑战赛上的大量实验表明,ReaTrack建立新性能标准,尤其在高阶推理子集上,RHOTA指标提升超三倍。数据集与代码将公开于https://github.com/chen-si-jia/ReaMOT。
原文摘要 · Abstract (English)
Referring Multi-Object Tracking (RMOT) aims to track targets specified by language instructions. However, existing RMOT paradigms heavily rely on explicit visual-textual matching and consequently fail to generalize to complex instructions that require logical reasoning. To overcome this, we propose Reasoning-based Multi-Object Tracking (ReaMOT), a novel task that elevates tracking to a cognitive level, requiring models to infer and track specific targets satisfying implicit constraints via logical reasoning. To advance this field, we construct the ReaMOT Challenge, a comprehensive benchmark featuring a tailored metric suite and a large scale dataset. This dataset comprises 1,156 language instructions, 423,359 image language pairs, and 869 distinct video sequences systematically categorized into six distinct evaluation scenarios, with over 75\% of the instructions dedicated to High Level Reasoning. Furthermore, recognizing that traditional trackers lack cognitive capacity while direct application of Large Vision-Language Model (LVLM) yields severe temporal inconsistencies, we propose ReaTrack. Driven by the insight to decouple high-level cognitive localization from low-level physical motion continuity, this training-free framework dynamically aligns the semantic detections of a Thinking-variant LVLM with the robust motion priors of SAM2. Extensive experiments on the ReaMOT Challenge benchmark demonstrate that ReaTrack establishes a new leading performance standard. Notably, it achieves a more than threefold improvement in RHOTA on the High Level Reasoning subset. Our dataset and code will be available at https://github.com/chen-si-jia/ReaMOT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。