让视觉语言模型学会分析物体运动轨迹与速度,提升时空推理能力。
ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in Vision-Language Models
- 构建含3D标注的时空推理数据集STKit,涵盖运动距离、速度等动态信息
- 通过4D重建自动生成伪标签,实现大规模无标注视频的数据扩展
- 新模型ST-VLM在多个基准上超越基线,支持复杂多步推理任务
时空推理在自动驾驶、体育分析等领域至关重要。尽管现有视觉语言模型(VLMs)在空间理解上取得进展,但对移动物体的运动学特征(如行进距离、速度)仍难以准确分析。为此,我们构建了包含真实视频与3D标注的时空推理数据集STKit及基准测试STKit-Bench,详细标注了物体运动动态:行进距离、速度、运动方向、物体间距离比较及相对运动方向。为扩展至无3D标签视频,我们提出基于4D重建的自动伪标签生成流程。基于此运动学指令微调数据,我们提出了ST-VLM,一个增强型时空推理视觉语言模型,在STKit-Bench上表现优异。进一步实验表明,该模型在跨领域、跨任务下具备强泛化能力,在ActivityNet、TVQA+等其他时空基准上也优于基线。通过融合学习到的时空推理能力,ST-VLM可实现复杂多步推理。
原文摘要 · Abstract (English)
Spatio-temporal reasoning is essential in understanding real-world environments in various fields, eg, autonomous driving and sports analytics. Recent advances have improved the spatial reasoning ability of Vision-Language Models (VLMs) by introducing large-scale data, but these models still struggle to analyze kinematic elements like traveled distance and speed of moving objects. To bridge this gap, we construct a spatio-temporal reasoning dataset and benchmark involving kinematic instruction tuning, referred to as STKit and STKit-Bench. They consist of real-world videos with 3D annotations, detailing object motion dynamics: traveled distance, speed, movement direction, inter-object distance comparisons, and relative movement direction. To further scale such data construction to videos without 3D labels, we propose an automatic pipeline to generate pseudo-labels using 4D reconstruction in real-world scale. With our kinematic instruction tuning data for spatio-temporal reasoning, we present ST-VLM, a VLM enhanced for spatio-temporal reasoning, which exhibits outstanding performance on STKit-Bench. Furthermore, we show that ST-VLM generalizes robustly across diverse domains and tasks, outperforming baselines on other spatio-temporal benchmarks (eg, ActivityNet, TVQA+). Finally, by integrating learned spatio-temporal reasoning with existing abilities, ST-VLM enables complex multi-step reasoning. Project page: https://ikodoh.github.io/ST-VLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。