用文字描述物体关系，提升大模型对3D场景的理解能力
Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptions
- 用自然语言描述物体属性和关系，增强3D理解
- 在5个基准数据集上均超越现有模型
- 无需额外标注，适合多任务场景理解
3D场景理解不仅需要识别物体，还需推理其空间与语义关系。当前3D场景-语言模型在关系理解上表现不足，尤其当仅依赖视觉嵌入时难以表达物体角色与交互。本文提出Descrip3D，通过自然语言显式编码物体间关系，为每个物体附加描述性文本，包含内在属性与上下文关系。该方法采用双层融合机制：嵌入级融合与提示级注入，实现统一推理，支持定位、描述生成、问答等任务，无需任务特定头或额外监督。在ScanRefer、Multi3DRefer、ScanQA、SQA3D、Scan2Cap五个基准数据集上，Descrip3D持续优于强基线模型，验证了语言引导关系表征在复杂室内场景理解中的有效性。代码与数据已开源。
原文摘要 · Abstract (English)
Understanding 3D scenes goes beyond simply recognizing objects; it requires reasoning about the spatial and semantic relationships between them. Current 3D scene-language models often struggle with this relational understanding, particularly when visual embeddings alone do not adequately convey the roles and interactions of objects. In this paper, we introduce Descrip3D, a novel and powerful framework that explicitly encodes the relationships between objects using natural language. Unlike previous methods that rely only on 2D and 3D embeddings, Descrip3D enhances each object with a textual description that captures both its intrinsic attributes and contextual relationships. These relational cues are incorporated into the model through a dual-level integration: embedding fusion and prompt-level injection. This allows for unified reasoning across various tasks such as grounding, captioning, and question answering, all without the need for task-specific heads or additional supervision. When evaluated on five benchmark datasets, including ScanRefer, Multi3DRefer, ScanQA, SQA3D, and Scan2Cap, Descrip3D consistently outperforms strong baseline models, demonstrating the effectiveness of language-guided relational representation for understanding complex indoor scenes. Our code and data are publicly available at https://github.com/jintangxue/Descrip3D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。