首个评测视频多跳空间推理能力的基准,揭示大模型在复杂空间推理中的短板。
Video-MSR: Benchmarking Multi-hop Spatial Reasoning Capabilities of MLLMs
- 构建四类任务评估视频中多步空间逻辑推理能力
- 20个主流模型在多跳任务上性能下降显著,易出现空间错乱和幻觉
- 推出专用微调数据集,使模型性能提升7.82个百分点
空间推理已成为多模态大模型的关键能力,但现有评测多聚焦单步感知判断任务,对需复杂视觉-空间逻辑链的场景关注不足。为此,我们提出Video-MSR,首个面向动态视频场景的多跳空间推理(MSR)评测基准。该基准通过四类任务系统评估:受限定位、链式指代检索、路径规划与反事实物理推断。包含3,052个高质量视频实例及4,993个问答对,基于可扩展的视觉锚定生成流程结合严格人工验证构建。对20个前沿多模态大模型的全面评估发现,尽管模型在表层感知表现良好,但在多跳推理任务中性能明显下滑,常出现空间错乱与幻觉。为缓解此问题,我们进一步构建了专用指令微调数据集MSR-9K,对Qwen-VL进行微调后,在Video-MSR上实现+7.82%的绝对性能提升。结果证明多跳空间指令数据的有效性,并确立Video-MSR作为未来研究的重要基础。代码与数据将开源于https://github.com/ruiz-nju/Video-MSR。
原文摘要 · Abstract (English)
Spatial reasoning has emerged as a critical capability for Multimodal Large Language Models (MLLMs), drawing increasing attention and rapid advancement. However, existing benchmarks primarily focus on single-step perception-to-judgment tasks, leaving scenarios requiring complex visual-spatial logical chains significantly underexplored. To bridge this gap, we introduce Video-MSR, the first benchmark specifically designed to evaluate Multi-hop Spatial Reasoning (MSR) in dynamic video scenarios. Video-MSR systematically probes MSR capabilities through four distinct tasks: Constrained Localization, Chain-based Reference Retrieval, Route Planning, and Counterfactual Physical Deduction. Our benchmark comprises 3,052 high-quality video instances with 4,993 question-answer pairs, constructed via a scalable, visually-grounded pipeline combining advanced model generation with rigorous human verification. Through a comprehensive evaluation of 20 state-of-the-art MLLMs, we uncover significant limitations, revealing that while models demonstrate proficiency in surface-level perception, they exhibit distinct performance drops in MSR tasks, frequently suffering from spatial disorientation and hallucination during multi-step deductions. To mitigate these shortcomings and empower models with stronger MSR capabilities, we further curate MSR-9K, a specialized instruction-tuning dataset, and fine-tune Qwen-VL, achieving a +7.82% absolute improvement on Video-MSR. Our results underscore the efficacy of multi-hop spatial instruction data and establish Video-MSR as a vital foundation for future research. The code and data will be available at https://github.com/ruiz-nju/Video-MSR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。