用3D分层场景图实现开放词汇的室内物体定位，支持复杂空间查询。
Open-Vocabulary Indoor Object Grounding with 3D Hierarchical Scene Graph
- 基于RGB-D帧构建分层场景图，融合多模态模型与传感器数据
- 在Habitat Matterport 3D数据集上实现高效场景理解与鲁棒物体定位
- 适合需要空间推理的智能机器人、AR/VR应用开发者
我们提出OVIGo-3DHSG方法——基于3D分层场景图的开放词汇室内物体定位。该方法利用一系列RGB-D帧，结合开放词汇基础模型与传感器数据处理，构建覆盖多楼层室内环境的分层场景图。该表示显式建模楼层、房间、位置与物体间的空间关系。为有效应对涉及其他物体空间参照的复杂查询，我们将分层场景图与大语言模型集成，实现多步推理，充分利用层间（如房间到物体）与层内（如物体到物体）连接，增强空间上下文理解能力。我们在Habitat Matterport 3D语义多楼层场景上评估了分层表示的语义与几何准确性。相比现有方法，本方法展现出更优的场景理解效率与鲁棒的物体定位性能。整体表明该方法在需空间推理的室内环境理解任务中具有强潜力。相关代码与资源见https://github.com/linukc/OVIGo-3DHSG。
原文摘要 · Abstract (English)
We propose OVIGo-3DHSG method - Open-Vocabulary Indoor Grounding of objects using 3D Hierarchical Scene Graph. OVIGo-3DHSG represents an extensive indoor environment over a Hierarchical Scene Graph derived from sequences of RGB-D frames utilizing a set of open-vocabulary foundation models and sensor data processing. The hierarchical representation explicitly models spatial relations across floors, rooms, locations, and objects. To effectively address complex queries involving spatial reference to other objects, we integrate the hierarchical scene graph with a Large Language Model for multistep reasoning. This integration leverages inter-layer (e.g., room-to-object) and intra-layer (e.g., object-to-object) connections, enhancing spatial contextual understanding. We investigate the semantic and geometry accuracy of hierarchical representation on Habitat Matterport 3D Semantic multi-floor scenes. Our approach demonstrates efficient scene comprehension and robust object grounding compared to existing methods. Overall OVIGo-3DHSG demonstrates strong potential for applications requiring spatial reasoning and understanding of indoor environments. Related materials can be found at https://github.com/linukc/OVIGo-3DHSG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。