arXiv:2506.04220cs.CV2025-06NeurIPS被引 7

用2D图像和标记实现多模态模型的3D空间推理,无需3D数据输入。

Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs

  • 通过鸟瞰图与物体标记构建结构化2D输入,引导模型进行空间推理。
  • 在200K条数据上微调后,模型在3D问答等任务上表现媲美闭源模型。
  • 适合研究视觉-语言融合、智能交互系统或自动驾驶场景理解的开发者。

提升多模态大模型(MLLMs)的空间推理能力对实现与3D环境的智能交互至关重要。现有方法多依赖显式的3D输入或专用架构,我们提出:能否仅使用感知生成的结构化2D表示,使MLLMs完成3D空间推理?为此,我们设计Struct2D,一个感知引导的提示框架,结合鸟瞰图(BEV)图像、物体标记及以物体为中心的元数据,必要时可加入自中心关键帧。利用Struct2D,我们对闭源MLLM(如GPT-o3)进行了零样本分析,发现其在结构化2D输入下展现出惊人空间推理能力,可有效完成相对方向估计与路径规划等任务。基于此,我们构建了Struct2D-Set,一个包含200K个细粒度问答对的大规模指令微调数据集,涵盖8类空间推理任务,由3D室内场景自动生成。我们在Qwen2.5VL上微调该数据集,取得在3D问答、密集描述和物体定位等多个基准上的竞争力表现。结果表明,结构化2D输入可有效连接感知与语言推理,无需显式3D输入。代码与数据集将开源,支持后续研究。

原文摘要 · Abstract (English)

Unlocking spatial reasoning in Multimodal Large Language Models (MLLMs) is crucial for enabling intelligent interaction with 3D environments. While prior efforts often rely on explicit 3D inputs or specialized model architectures, we ask: can MLLMs reason about 3D space using only structured 2D representations derived from perception? We introduce Struct2D, a perception-guided prompting framework that combines bird's-eye-view (BEV) images with object marks and object-centric metadata, optionally incorporating egocentric keyframes when needed. Using Struct2D, we conduct an in-depth zero-shot analysis of closed-source MLLMs (e.g., GPT-o3) and find that they exhibit surprisingly strong spatial reasoning abilities when provided with structured 2D inputs, effectively handling tasks such as relative direction estimation and route planning. Building on these insights, we construct Struct2D-Set, a large-scale instruction tuning dataset with 200K fine-grained QA pairs across eight spatial reasoning categories, generated automatically from 3D indoor scenes. We fine-tune an open-source MLLM (Qwen2.5VL) on Struct2D-Set, achieving competitive performance on multiple benchmarks, including 3D question answering, dense captioning, and object grounding. Our approach demonstrates that structured 2D inputs can effectively bridge perception and language reasoning in MLLMs-without requiring explicit 3D representations as input. We will release both our code and dataset to support future research.

空间推理多模态模型2D输入结构化提示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。