arXiv:2507.16524cs.CVcs.AI2025-07中稿 · ICME2025被引 7

提升3D视觉语言模型的空间感知能力,让AI更懂物体位置关系。

Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models

  • 采用渐进式空间感知架构,逐步捕捉场景中物体的位置信息。
  • 在3D物体距离测量与布局编辑任务上达到当前最佳表现。
  • 适合研究3D场景理解、机器人导航的开发者使用。

大语言模型(LLM)在扩展至3D视觉语言任务方面开启了新纪元。然而,现有3D多模态大模型(MLLM)通常依赖压缩整体3D场景信息或分割独立物体来完成任务,导致空间感知能力受限。为此,我们提出Spatial 3D-LLM,一种专为增强3D视觉语言任务空间感知而设计的模型,通过丰富3D场景的空间嵌入实现。该模型结合了LLM主干网络与渐进式空间感知机制,随着感知范围扩大逐步捕获空间信息,生成带位置信息的3D场景嵌入作为视觉提示。此外,我们提出了两个新任务:3D物体距离测量和3D布局编辑,并构建了名为MODEL的3D指令数据集以评估模型的空间感知能力。实验表明,Spatial 3D-LLM在多种3D视觉语言任务中均达到领先性能,验证了其渐进式空间感知机制对挖掘深层空间信息的有效性。代码已开源:https://github.com/bjshuyuan/Spatial-3D-LLM。

原文摘要 · Abstract (English)

New era has unlocked exciting possibilities for extending Large Language Models (LLMs) to tackle 3D vision-language tasks. However, most existing 3D multimodal LLMs (MLLMs) rely on compressing holistic 3D scene information or segmenting independent objects to perform these tasks, which limits their spatial awareness due to insufficient representation of the richness inherent in 3D scenes. To overcome these limitations, we propose Spatial 3D-LLM, a 3D MLLM specifically designed to enhance spatial awareness for 3D vision-language tasks by enriching the spatial embeddings of 3D scenes. Spatial 3D-LLM integrates an LLM backbone with a progressive spatial awareness scheme that progressively captures spatial information as the perception field expands, generating location-enriched 3D scene embeddings to serve as visual prompts. Furthermore, we introduce two novel tasks: 3D object distance measurement and 3D layout editing, and construct a 3D instruction dataset, MODEL, to evaluate the model's spatial awareness capabilities. Experimental results demonstrate that Spatial 3D-LLM achieves state-of-the-art performance across a wide range of 3D vision-language tasks, revealing the improvements stemmed from our progressive spatial awareness scheme of mining more profound spatial information. Our code is available at https://github.com/bjshuyuan/Spatial-3D-LLM.

3D感知视觉语言模型空间理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。