arXiv:2505.23747cs.CVcs.AI2025-05NeurIPS被引 224

仅用2D图像视频提升模型空间推理能力,无需3D数据。

Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence

论文配图:Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence
图 1 · 摘自论文原文
  • 用2D+3D结构先验双编码器融合视觉语义与空间结构。
  • 在多个数据集上超越现有方法,尤其在空间推理任务中提升显著。
  • 适合需要从2D输入理解空间关系的场景,如自动驾驶、机器人导航。

多模态大模型在2D视觉任务上表现优异,但空间智能仍面临挑战。现有3D MLLM依赖额外3D或2.5D数据,难以处理仅含2D输入(如图像、视频)的场景。本文提出Spatial-MLLM,一种仅基于2D观测的视觉空间推理新框架。不同于传统视频MLLM使用的以语义理解优化的CLIP编码器,我们的核心思路是利用前馈视觉几何基础模型中的强结构先验。设计双编码器架构:一个预训练2D视觉编码器提取语义特征,一个由视觉几何模型主干初始化的3D空间编码器提取3D结构特征;通过连接器将两者融合为统一视觉令牌,增强空间理解。此外,提出空间感知帧采样策略,在推理时选择空间信息丰富的视频帧,确保在有限上下文长度下仍聚焦关键帧。除架构改进外,整合多源数据构建训练集,并采用监督微调与GRPO进行训练。大量实验证明,Spatial-MLLM在多种真实世界数据集上的视觉空间理解与推理任务中达到领先水平。

原文摘要 · Abstract (English)

Recent advancements in Multimodal Large Language Models (MLLMs) have significantly enhanced performance on 2D visual tasks. However, improving their spatial intelligence remains a challenge. Existing 3D MLLMs always rely on additional 3D or 2.5D data to incorporate spatial awareness, restricting their utility in scenarios with only 2D inputs, such as images or videos. In this paper, we present Spatial-MLLM, a novel framework for visual-based spatial reasoning from purely 2D observations. Unlike conventional video MLLMs which rely on CLIP-based visual encoders optimized for semantic understanding, our key insight is to unleash the strong structure prior from the feed-forward visual geometry foundation model. Specifically, we propose a dual-encoder architecture: a pretrained 2D visual encoder to extract semantic features, and a 3D spatial encoder-initialized from the backbone of the visual geometry model-to extract 3D structure features. A connector then integrates both features into unified visual tokens for enhanced spatial understanding. Furthermore, we propose a space-aware frame sampling strategy at inference time, which selects the spatially informative frames of a video sequence, ensuring that even under limited token length, the model focuses on frames critical for spatial reasoning. Beyond architecture improvements, we construct a training dataset from multiple sources and train the model on it using supervised fine-tuning and GRPO. Extensive experiments on various real-world datasets demonstrate that Spatial-MLLM achieves state-of-the-art performance in a wide range of visual-based spatial understanding and reasoning tasks. Project page: https://diankun-wu.github.io/Spatial-MLLM/.

空间推理多模态模型2D到3D视觉几何

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。