arXiv:2509.16721cs.CVcs.AI2025-09被引 1

将3D场景自动解析为自然语言描述,助力智能体理解复杂环境。

Text-Scene: A Scene-to-Language Parsing Framework for 3D Scene Understanding

  • 结合几何分析与多模态大模型,自动提取物体属性与空间关系。
  • 在3174个任务的InPlan3D基准上验证,生成描述准确且利于下游任务。
  • 适合做3D场景理解、智能体导航与具身AI研究者使用。

让智能体理解并交互于复杂3D场景是具身人工智能的核心挑战。尽管多模态大模型在2D图像理解上取得显著进展,但将其能力扩展至3D场景仍面临两大难题:1)3D环境包含空间关系、可操作性、物理特性、布局等更丰富概念;2)缺乏大规模3D视觉-语言数据集。本文提出Text-Scene框架,能自动将3D场景解析为文本描述。给定一个3D场景,该模型识别物体属性与空间关系,并生成连贯的整体描述,实现无需人工干预的3D观测到语言的桥梁。通过融合几何分析与多模态大模型(MLLMs),Text-Scene产出的描述兼具准确性、细节性与人类可读性,涵盖物体级细节与全局上下文。实验结果表明,其文本解析能忠实表征3D场景,并提升下游任务性能。为此,我们构建了InPlan3D——一个包含636个室内场景、3174个长期规划任务的综合性3D任务规划评测基准,用于评估MLLM的推理能力。本工作强调清晰与可访问性,旨在通过语言使3D场景内容更易理解。代码与数据集将公开发布。

原文摘要 · Abstract (English)

Enabling agents to understand and interact with complex 3D scenes is a fundamental challenge for embodied artificial intelligence systems. While Multimodal Large Language Models (MLLMs) have achieved significant progress in 2D image understanding, extending such capabilities to 3D scenes remains difficult: 1) 3D environment involves richer concepts such as spatial relationships, affordances, physics, layout, and so on, 2) the absence of large-scale 3D vision-language datasets has posed a significant obstacle. In this paper, we introduce Text-Scene, a framework that automatically parses 3D scenes into textual descriptions for scene understanding. Given a 3D scene, our model identifies object attributes and spatial relationships, and then generates a coherent summary of the whole scene, bridging the gap between 3D observation and language without requiring human-in-the-loop intervention. By leveraging both geometric analysis and MLLMs, Text-Scene produces descriptions that are accurate, detailed, and human-interpretable, capturing object-level details and global-level context. Experimental results on benchmarks demonstrate that our textual parses can faithfully represent 3D scenes and benefit downstream tasks. To evaluate the reasoning capability of MLLMs, we present InPlan3D, a comprehensive benchmark for 3D task planning, consisting of 3174 long-term planning tasks across 636 indoor scenes. We emphasize clarity and accessibility in our approach, aiming to make 3D scene content understandable through language. Code and datasets will be released.

3D理解多模态场景解析具身AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。