让3D高斯点实现跨场景开放词汇语义分割,通用性更强。
OVGaussian: Generalizable 3D Gaussian Segmentation with Open Vocabularies
- 用3D神经网络预测高斯点语义,生成一致的多视角语义图。
- 在SegGaussian数据集上训练,跨场景、跨领域性能显著优于基线。
- 适合做开放词汇3D场景理解的研究者与开发者参考。
基于3D高斯表示的开放词汇场景理解受到广泛关注。然而,现有方法大多将大型2D视觉模型的知识逐场景迁移至3D高斯表示,限制了其在训练场景内的开放词汇查询能力,缺乏对新场景的泛化能力。本文提出 extbf{OVGaussian},一种基于3D高斯表示的可泛化的开放词汇3D语义分割框架。首先构建大规模3D场景数据集SegGaussian,包含高斯点和多视角图像的细粒度语义与实例标注。为促进跨场景语义泛化,提出可泛化的语义光栅化(GSR),利用3D神经网络学习并预测每个3D高斯点的语义属性,该属性可渲染为多视图一致的2D语义图。随后提出跨模态一致性学习(CCL)框架,利用SegGaussian中2D图像与3D高斯点的开放词汇标注,训练具备跨高斯3D场景开放词汇语义分割能力的3D神经网络。实验表明,OVGaussian在跨场景、跨域及新视角下均表现优异,显著超越基线方法。代码与数据集将公开。 (https://github.com/runnanchen/OVGaussian)
原文摘要 · Abstract (English)
Open-vocabulary scene understanding using 3D Gaussian (3DGS) representations has garnered considerable attention. However, existing methods mostly lift knowledge from large 2D vision models into 3DGS on a scene-by-scene basis, restricting the capabilities of open-vocabulary querying within their training scenes so that lacking the generalizability to novel scenes. In this work, we propose \textbf{OVGaussian}, a generalizable \textbf{O}pen-\textbf{V}ocabulary 3D semantic segmentation framework based on the 3D \textbf{Gaussian} representation. We first construct a large-scale 3D scene dataset based on 3DGS, dubbed \textbf{SegGaussian}, which provides detailed semantic and instance annotations for both Gaussian points and multi-view images. To promote semantic generalization across scenes, we introduce Generalizable Semantic Rasterization (GSR), which leverages a 3D neural network to learn and predict the semantic property for each 3D Gaussian point, where the semantic property can be rendered as multi-view consistent 2D semantic maps. In the next, we propose a Cross-modal Consistency Learning (CCL) framework that utilizes open-vocabulary annotations of 2D images and 3D Gaussians within SegGaussian to train the 3D neural network capable of open-vocabulary semantic segmentation across Gaussian-based 3D scenes. Experimental results demonstrate that OVGaussian significantly outperforms baseline methods, exhibiting robust cross-scene, cross-domain, and novel-view generalization capabilities. Code and the SegGaussian dataset will be released. (https://github.com/runnanchen/OVGaussian).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。