无需2D标签,用自举法实现3D高斯点云的语义一致性理解
Bootstraping Clustering of Gaussians for View-consistent 3D Scene Understanding
- 引入耦合语义场,让每个高斯点同时拥有语义和视图一致的实例编号
- 两步交替优化:3D语义聚类引导实例分割,实例又反哺稳定语义注入
- 在多个数据集上达到顶尖性能,且无需复杂标注流程,适合真实场景应用
将语义信息融入3D高斯溅射(3DGS)近年受到广泛关注。现有方法通常从2D基础模型(如CLIP和SAM)中蒸馏3D语义特征,以支持新视角分割与语义理解,但过度依赖2D监督会破坏跨视角语义一致性,并需复杂的数据预处理,阻碍了视图一致的场景理解。本文提出FreeGS,一种无需2D标签的无监督语义嵌入3DGS框架,实现了视图一致的3D场景理解。我们引入身份耦合语义场(IDSF),使每个高斯点同时捕捉语义表示和视图一致的实例索引。通过两步交替优化策略:3D语义帮助提取连贯实例,而生成的实例又规范来自2D空间的稳定语义注入。此外,采用2D-3D联合对比损失,在自举过程中增强视图一致的3D几何与丰富语义之间的互补性,使FreeGS在新视角语义分割、物体选择和3D物体检测等任务上表现统一。在LERF-Mask、3D-OVS和ScanNet数据集上的大量实验表明,FreeGS性能媲美当前最优方法,同时避免复杂的预处理工作量。代码已公开于https://github.com/wb014/FreeGS。
原文摘要 · Abstract (English)
Injecting semantics into 3D Gaussian Splatting (3DGS) has recently garnered significant attention. While current approaches typically distill 3D semantic features from 2D foundational models (e.g., CLIP and SAM) to facilitate novel view segmentation and semantic understanding, their heavy reliance on 2D supervision can undermine cross-view semantic consistency and necessitate complex data preparation processes, therefore hindering view-consistent scene understanding. In this work, we present FreeGS, an unsupervised semantic-embedded 3DGS framework that achieves view-consistent 3D scene understanding without the need for 2D labels. Instead of directly learning semantic features, we introduce the IDentity-coupled Semantic Field (IDSF) into 3DGS, which captures both semantic representations and view-consistent instance indices for each Gaussian. We optimize IDSF with a two-step alternating strategy: semantics help to extract coherent instances in 3D space, while the resulting instances regularize the injection of stable semantics from 2D space. Additionally, we adopt a 2D-3D joint contrastive loss to enhance the complementarity between view-consistent 3D geometry and rich semantics during the bootstrapping process, enabling FreeGS to uniformly perform tasks such as novel-view semantic segmentation, object selection, and 3D object detection. Extensive experiments on LERF-Mask, 3D-OVS, and ScanNet datasets demonstrate that FreeGS performs comparably to state-of-the-art methods while avoiding the complex data preprocessing workload. Our code is publicly available at https://github.com/wb014/FreeGS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。