用结构化程序访问3D空间记忆,提升视觉语言模型的空间推理可靠性
Reasmory: 3D Reconstruction as Explicit Memory for VLMs Spatial Reasoning

- 将3D重建结果作为显式空间记忆,用专用语言约束查询与操作
- 在多视角图像和视频任务上提升6%~18%,超越GPT-5-mini等强基线
- 适合需要精确空间理解的场景,如视角判断、距离估计
视觉语言模型(VLMs)展现出一定的空间推理能力,但在需要精确空间理解的任务(如视角推理、方向比较、距离估计)中仍不可靠。多视角图像和单目视频中的空间线索常稀疏且分散于冗余观测中,难以组织利用。基于重建的视觉基础模型(VFMs)可将这些观测聚合为显式空间记忆,如点云。然而,直接暴露重建模型作为自由工具存在脆弱性:VLM可能误用工具、跳过必要空间变换或错误使用中间结果。我们提出「Reasmory」框架,将空间推理建模为对重构空间记忆的结构化程序执行。Reasmory构建显式3D记忆,融合语义标注的3D物体实例,并引入轻量级领域特定语言(DSL),约束VLM如何查询对象与相机、转换视角、渲染观察。生成的程序在执行前经解析与验证,相比无约束工具调用更可靠。在多视角图像与视频空间推理基准上的实验显示,性能相较强基线(包括GPT-5-mini与Gemini-3-flash)稳定提升6%~18%,表明显式3D记忆需通过受控、验证的操作访问才最有效。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) exhibit emerging spatial reasoning capabilities, yet they remain unreliable on tasks requiring precise spatial understanding, such as viewpoint reasoning, directional comparison, and distance estimation. In multi-view images and monocular videos, relevant spatial cues are often sparse and distributed across redundant observations, making them difficult to organize and exploit. Reconstruction-based Vision Foundation Models (VFMs) offer a natural way to aggregate such observations into explicit spatial memory, such as point clouds. However, simply exposing reconstruction models as free-form tools is brittle, VLMs may invoke tools incorrectly, skip required spatial transformations, or misuse intermediate results. We propose \textbf{Reasmory}, a framework that formulates spatial reasoning as structured program execution over reconstructed spatial memory. Reasmory constructs explicit 3D memory, augments it with semantically grounded 3D object instances, and introduces a lightweight Domain-Specific Language (DSL) that constrains how VLMs query objects and cameras, transform viewpoints, and render observations during reasoning. Generated programs are parsed and validated before execution, enabling more reliable interaction with spatial memory than unconstrained tool use. Experiments on multi-view image and video spatial reasoning benchmarks show consistent gains of 6--18\% over strong baselines, including GPT-5-mini and Gemini-3-flash, indicating that explicit 3D memory is most useful when accessed through constrained, validated operations rather than free-form tool calls.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。