端到端追踪视频中多个物体,用语言描述精准定位。
STORM: End-to-End Referring Multi-Object Tracking in Videos

- 统一框架同时完成物体定位与追踪,无需外部检测器。
- 在三个基准上达到最优,复杂场景下仍保持稳定表现。
- 适合需要多物体语言追踪的视觉理解研究者。
指称多物体追踪(RMOT)是将视频中所有语义匹配给定文本查询或指称表达的物体关联起来的任务。现有方法将物体定位与追踪分作独立模块,受限于训练视频少、标注模糊及领域受限,性能有限。本文提出STORM,一种端到端多模态大模型,统一框架内联合执行定位与追踪,实现外观、运动与语言的连贯推理。为提升数据效率,提出任务组合学习策略(TCL),将RMOT拆解为图像定位与物体追踪两个子任务,利用数据丰富的子任务学习结构化时空推理。进一步构建了STORM-Bench数据集,通过自底向上的标注流程生成准确轨迹和多样、清晰的指称表达。大量实验表明,STORM在图像定位、单物体追踪及RMOT基准上均达当前最优,展现出强泛化能力与复杂真实场景下的鲁棒时空定位能力。相关数据集已开源:https://github.com/amazon-science/storm-referring-multi-object-grounding。
原文摘要 · Abstract (English)
Referring multi-object tracking (RMOT) is a task of associating all the objects in a video that semantically match with given textual queries or referring expressions. Existing RMOT approaches decompose object grounding and tracking into separated modules and exhibit limited performance due to the scarcity of training videos, ambiguous annotations, and restricted domains. In this work, we introduce STORM, an end-to-end MLLM that jointly performs grounding and tracking within a unified framework, eliminating external detectors and enabling coherent reasoning over appearance, motion, and language. To improve data efficiency, we propose a task-composition learning (TCL) strategy that decomposes RMOT into image grounding and object tracking, allowing STORM to leverage data-rich sub-tasks and learn structured spatial--temporal reasoning. We further construct STORM-Bench, a new RMOT dataset with accurate trajectories and diverse, unambiguous referring expressions generated through a bottom-up annotation pipeline. Extensive experiments show that STORM achieves state-of-the-art performance on image grounding, single-object tracking, and RMOT benchmarks, demonstrating strong generalization and robust spatial--temporal grounding in complex real-world scenarios. STORM-Bench is released at https://github.com/amazon-science/storm-referring-multi-object-grounding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。