用混合3D高斯点云实现开放词汇场景理解，提升物体级识别精度
OpenGS-Fusion: Open-Vocabulary Dense Mapping with Hybrid 3D Gaussian Splatting for Refined Object-Level Understanding
- 融合3D高斯与截断符号距离场，实时融合语义特征
- 语言引导自适应阈值使3D mIoU提升17%
- 适合需要开放查询的AR/VR与机器人应用
近期3D场景理解进展推动了基于开放词汇查询的场景交互，尤其在虚拟现实、增强现实和机器人领域。然而现有方法受限于固定离线流程，难以对开放式查询提供精确的3D物体级理解。本文提出OpenGS-Fusion，一种创新的开放词汇密集映射框架，通过结合3D高斯表示与截断符号距离场，实现实时无损语义特征融合。同时引入新型多模态语言引导自适应阈值方法（MLLM-Assisted Adaptive Thresholding），通过动态调整相似度阈值，使3D mIoU相比固定阈值策略提升17%。大量实验表明，该方法在3D物体理解与场景重建质量上优于现有方法，并在语言引导场景交互中表现出色。代码已公开于 https://young-bit.github.io/opengs-fusion.github.io/。
原文摘要 · Abstract (English)
Recent advancements in 3D scene understanding have made significant strides in enabling interaction with scenes using open-vocabulary queries, particularly for VR/AR and robotic applications. Nevertheless, existing methods are hindered by rigid offline pipelines and the inability to provide precise 3D object-level understanding given open-ended queries. In this paper, we present OpenGS-Fusion, an innovative open-vocabulary dense mapping framework that improves semantic modeling and refines object-level understanding. OpenGS-Fusion combines 3D Gaussian representation with a Truncated Signed Distance Field to facilitate lossless fusion of semantic features on-the-fly. Furthermore, we introduce a novel multimodal language-guided approach named MLLM-Assisted Adaptive Thresholding, which refines the segmentation of 3D objects by adaptively adjusting similarity thresholds, achieving an improvement 17\% in 3D mIoU compared to the fixed threshold strategy. Extensive experiments demonstrate that our method outperforms existing methods in 3D object understanding and scene reconstruction quality, as well as showcasing its effectiveness in language-guided scene interaction. The code is available at https://young-bit.github.io/opengs-fusion.github.io/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。