解决3D场景中视角依赖语义问题，提升语言驱动的3D理解能力。
Tackling View-Dependent Semantics in 3D Language Gaussian Splatting
- 将3D场景分解为物体，建立跨视角语义关联。
- 通过聚类和重加权构建多视角语义融合表示，提升理解精度。
- 在LERF-OVS数据集上比当前最优方法提升18.7% mIoU，适合3D视觉与语言研究者。
3D高斯点云（3D-GS）技术已实现从RGB图像高质量重建3D场景。许多研究将其扩展至语言驱动的开放词汇场景理解。然而，现有方法通常简单将2D语义特征投影到3D高斯点，忽略了2D与3D理解间的根本差异：同一3D物体在不同视角下可能呈现不同语义——我们称之为视角依赖语义。为此，我们提出LaGa（Language Gaussians），通过将3D场景分解为物体，建立跨视角语义连接；再通过聚类语义描述子并基于多视角语义重加权，构建视图聚合的语义表示。大量实验表明，LaGa能有效捕捉视角依赖语义的关键信息，实现更全面的3D场景理解。值得注意的是，在相同设置下，其在LERF-OVS数据集上相比先前最优方法取得+18.7% mIoU的显著提升。代码已公开于：https://github.com/SJTU-DeepVisionLab/LaGa。
原文摘要 · Abstract (English)
Recent advancements in 3D Gaussian Splatting (3D-GS) enable high-quality 3D scene reconstruction from RGB images. Many studies extend this paradigm for language-driven open-vocabulary scene understanding. However, most of them simply project 2D semantic features onto 3D Gaussians and overlook a fundamental gap between 2D and 3D understanding: a 3D object may exhibit various semantics from different viewpoints--a phenomenon we term view-dependent semantics. To address this challenge, we propose LaGa (Language Gaussians), which establishes cross-view semantic connections by decomposing the 3D scene into objects. Then, it constructs view-aggregated semantic representations by clustering semantic descriptors and reweighting them based on multi-view semantics. Extensive experiments demonstrate that LaGa effectively captures key information from view-dependent semantics, enabling a more comprehensive understanding of 3D scenes. Notably, under the same settings, LaGa achieves a significant improvement of +18.7% mIoU over the previous SOTA on the LERF-OVS dataset. Our code is available at: https://github.com/SJTU-DeepVisionLab/LaGa.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。