无需训练即可实现多对象语义追踪，利用视觉语言模型生成描述与交互识别。
Training-Free Semantic Multi-Object Tracking with Vision-Language Models
- 组合预训练检测、分割追踪与视频语言模型，零训练构建追踪流水线。
- 在BenSMOT上达到顶尖追踪性能，且摘要与实例描述质量优于已有方法。
- 适用于快速适配新模型和新交互场景，尤其适合研究语义追踪的开发者。
语义多目标追踪（SMOT）通过视频摘要、实例级描述和交互标签等语义输出，将追踪从轨迹延伸至动态场景的人类可理解描述。现有系统依赖端到端训练，受制于昂贵标注，难以快速适应新基础模型和新交互。本文提出无训练的TF-SMOT框架，整合D-FINE与可提示SAM2分割追踪器生成时序一致的轨迹片段，利用轮廓定位结合InternVideo2.5生成视频摘要与实例描述，并通过基于词典的语义检索与LLM消歧，将提取的交互谓词对齐至BenSMOT WordNet同义词集。在BenSMOT数据集上，TF-SMOT在SMOT设置下达到最先进追踪表现，且摘要与描述质量超越此前方法。然而，在细粒度、长尾的WordNet标签空间下严格匹配评估中，交互识别仍具挑战；分析表明，语义重叠与标签粒度显著影响评测结果。
原文摘要 · Abstract (English)
Semantic Multi-Object Tracking (SMOT) extends multi-object tracking with semantic outputs such as video summaries, instance-level captions, and interaction labels, aiming to move from trajectories to human-interpretable descriptions of dynamic scenes. Existing SMOT systems are trained end-to-end, coupling progress to expensive supervision, limiting the ability to rapidly adapt to new foundation models and new interactions. We propose TF-SMOT, a training-free SMOT pipeline that composes pretrained components for detection, mask-based tracking, and video-language generation. TF-SMOT combines D-FINE and the promptable SAM2 segmentation tracker to produce temporally consistent tracklets, uses contour grounding to generate video summaries and instance captions with InternVideo2.5, and aligns extracted interaction predicates to BenSMOT WordNet synsets via gloss-based semantic retrieval with LLM disambiguation. On BenSMOT, TF-SMOT achieves state-of-the-art tracking performance within the SMOT setting and improves summary and caption quality compared to prior art. Interaction recognition, however, remains challenging under strict exact-match evaluation on the fine-grained and long-tailed WordNet label space; our analysis and ablations indicate that semantic overlap and label granularity substantially affect measured performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。