让大模型用代码操控3D工具，零样本完成空间推理
pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoning
- 通过生成Python代码调用3D工具链，构建可探索的三维场景
- 在MindCube上比GPT-4.1-mini高12.94%，无需微调
- 适合需要3D空间理解的机器人导航、虚拟现实等场景
多模态大语言模型虽具备通用感知与推理能力，但在需要理解三维世界的空间任务上仍表现不佳。为此，我们提出pySpatial，一种视觉编程框架，使MLLMs可通过生成Python代码与空间工具交互。给定图像序列和自然语言查询，模型组合调用3D重建、相机位姿恢复、新视角渲染等操作，将原始2D输入转化为可探索的3D场景，使大模型能基于结构化空间表示进行显式推理。pySpatial无需梯度微调，完全在零样本设置下运行。在MindCube和Omni3D-Bench两个挑战性基准上的实验表明，该框架持续超越强基线模型；例如，在MindCube上比GPT-4.1-mini高出12.94%。此外，真实室内导航实验中，机器人成功利用pySpatial生成的路径规划穿越复杂环境，验证了方法的实际有效性。
原文摘要 · Abstract (English)
Multi-modal Large Language Models (MLLMs) have demonstrated strong capabilities in general-purpose perception and reasoning, but they still struggle with tasks that require spatial understanding of the 3D world. To address this, we introduce pySpatial, a visual programming framework that equips MLLMs with the ability to interface with spatial tools via Python code generation. Given an image sequence and a natural-language query, the model composes function calls to spatial tools including 3D reconstruction, camera-pose recovery, novel-view rendering, etc. These operations convert raw 2D inputs into an explorable 3D scene, enabling MLLMs to reason explicitly over structured spatial representations. Notably, pySpatial requires no gradient-based fine-tuning and operates in a fully zero-shot setting. Experimental evaluations on the challenging MindCube and Omni3D-Bench benchmarks demonstrate that our framework pySpatial consistently surpasses strong MLLM baselines; for instance, it outperforms GPT-4.1-mini by 12.94% on MindCube. Furthermore, we conduct real-world indoor navigation experiments where the robot can successfully traverse complex environments using route plans generated by pySpatial, highlighting the practical effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。