让大模型学会在动态3D场景中全局推理物体位置关系。
Video Spatial Reasoning with Object-Centric 3D Rollout
- 通过扰动物体3D几何结构,迫使模型依赖整体场景而非局部提示。
- 30亿参数模型在VSI-Bench上达47.5%准确率,优于多个70亿参数模型。
- 适合需要精准空间理解的视频分析任务,如机器人导航、自动驾驶。
多模态大语言模型在视觉-语言理解方面取得显著进展,但实现鲁棒的视频空间推理——即在动态3D场景中理解物体位置、朝向及相互关系——仍是未解难题。现有方法主要依赖空间锚定的监督微调或强化学习,但模型常出现查询锁定现象,仅关注提示中明确提及的物体,忽略关键上下文线索。为此,我们提出基于对象中心的3D滚动(OCR)策略,在训练中对选定物体的3D几何施加结构化扰动。通过削弱特定物体的视觉线索并将修改后的几何投影至2D空间,促使模型在全场景范围内进行整体推理。我们还设计了一种基于滚动的训练流程,联合使用原始视频与区域噪声视频以优化空间推理轨迹。实验表明,我们的30亿参数模型在VSI-Bench上达到47.5%准确率,超越多个70亿参数基线模型。消融实验验证了OCR相较先前滚动策略(如T-GRPO、NoisyRollout)的优越性。
原文摘要 · Abstract (English)
Recent advances in Multi-modal Large Language Models (MLLMs) have showcased remarkable capabilities in vision-language understanding. However, enabling robust video spatial reasoning-the ability to comprehend object locations, orientations, and inter-object relationships in dynamic 3D scenes-remains a key unsolved challenge. Existing approaches primarily rely on spatially grounded supervised fine-tuning or reinforcement learning, yet we observe that such models often exhibit query-locked reasoning, focusing narrowly on objects explicitly mentioned in the prompt while ignoring critical contextual cues. To address this limitation, we propose Object-Centric 3D Rollout (OCR), a novel strategy that introduces structured perturbations to the 3D geometry of selected objects during training. By degrading object-specific visual cues and projecting the altered geometry into 2D space, OCR compels the model to reason holistically across the entire scene. We further design a rollout-based training pipeline that jointly leverages vanilla and region-noisy videos to optimize spatial reasoning trajectories. Experiments demonstrate state-of-the-art performance: our 3B-parameter model achieves 47.5% accuracy on VSI-Bench, outperforming several 7B baselines. Ablations confirm OCR's superiority over prior rollout strategies (e.g., T-GRPO, NoisyRollout).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。