用大模型让追踪系统能生成语义理解,突破传统定位局限。
Generative Semantic Multi-Object Tracking: A Large-Scale Benchmark and an MLLM-Driven Reasoning Framework
- 将多目标追踪转为开放生成任务,用大模型推理动态场景
- 在高密度双流数据集上实现几何追踪最优,语义推理显著提升
- 适合关注视频理解、大模型应用的研究者
语义多目标追踪(SMOT)正从纯几何定位转向全面视频理解。但现有方法多依赖封闭集标签和碎片化感知流程,限制了多模态大模型在动态场景中的应用。本文将SMOT升级为开放式生成推理任务,提出大型基准Grand-SMOT,包含高密度、双流叙事数据,显式分离个体微观动态与宏观环境上下文,解决以往数据集语义匮乏问题。同时提出统一的MLLM驱动框架LLMTrack,采用可验证的‘宏观理解优先’机制,通过时空融合模块将离散轨迹压缩为连续语义标记,有效抑制长序列追踪中的时间幻觉。基于新提出的解耦评估协议,实验表明LLMTrack在几何追踪鲁棒性上达到当前最优,且在生成式语义推理方面实现质的飞跃。代码与数据集已开源:https://github.com/liaopan-lp/LLMTrack-GrandSMOT。
原文摘要 · Abstract (English)
Semantic Multi-Object Tracking (SMOT) is evolving from purely geometric localization toward comprehensive video understanding. However, existing paradigms predominantly rely on closed-set interaction tags and fragmented perception pipelines, creating a bottleneck that prevents the full utilization of Multi-modal Large Language Models (MLLMs) for dynamic scenes. In this paper, we elevate SMOT from rigid classification to an open-ended generative reasoning task. To support this paradigm shift, we introduce Grand-SMOT, a large-scale benchmark featuring high-density, dual-stream narratives. This dataset explicitly decouples micro-level individual dynamics from macro-level environmental contexts, directly resolving the semantic scarcity of prior tracking datasets. Furthermore, we propose LLMTrack, a unified MLLM-driven framework for dynamic SMOT. Guided by a verifiable ``\textit{Macro-Understanding-First}'' mechanism, LLMTrack employs a Spatio-Temporal Fusion Module to compress discrete geometric trajectories into continuous semantic tokens, effectively suppressing temporal hallucinations in long-sequence tracking. Extensive experiments, utilizing a novel decoupled evaluation protocol, validate that LLMTrack achieves state-of-the-art geometric tracking robustness while delivering a qualitative leap in generative semantic reasoning. The code and datasets are publicly available at https://github.com/liaopan-lp/LLMTrack-GrandSMOT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。