让大模型直接在3D空间地图上推理，提升多视角图像理解能力
Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps
- 构建显式3D记忆地图，每个视觉标记都有三维位置和语义信息
- 在多个空间推理基准上达到当前最优性能
- 适合需要精准空间理解的机器人、自动驾驶等场景
从多视角图像中实现精确的空间理解仍是多模态大语言模型（MLLM）的核心挑战，因其视觉表征主要为语义信息，缺乏明确的几何基础。现有方法虽通过视觉几何模型为视觉标记添加几何线索，但模型仍需隐式推断场景的3D结构，限制了空间推理能力。为此，我们提出Cog3DMap框架，通过多视角图像反复构建一个显式的3D记忆地图，其中每个标记均在三维空间中定位，并兼具语义与几何信息。将这些标记输入MLLM后，可直接在结构化3D地图上进行推理，在多个空间推理基准上取得当前最优表现。代码将公开。
原文摘要 · Abstract (English)
Precise spatial understanding from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs), as their visual representations are predominantly semantic and lack explicit geometric grounding. While existing approaches augment visual tokens with geometric cues from visual geometry models, their MLLM is still required to implicitly infer the underlying 3D structure of the scene from these augmented tokens, limiting its spatial reasoning capability. To address this issue, we introduce Cog3DMap, a framework that recurrently constructs an explicit 3D memory from multi-view images, where each token is grounded in 3D space and possesses both semantic and geometric information. By feeding these tokens into the MLLM, our framework enables direct reasoning over a spatially structured 3D map, achieving state-of-the-art performance on various spatial reasoning benchmarks. Code will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。