用高斯点云分离3D分割与语义识别,实现无需重训练的开放词汇3D分割。
DCSEG: Decoupled 3D Open-Set Segmentation using Gaussian Splatting

- 分两步:先用3D高斯重建场景并生成粗略掩码,再匹配2D模型的语义标签。
- 在合成与真实室内数据集上,mIoU和mAcc均优于基于NeRF的方法,尤其对长尾类别提升明显。
- 框架模块化,可灵活替换2D/3D组件,适合需要快速适配新模型的机器人与AR应用。
开放词汇3D分割对机器人及增强/虚拟现实应用具有重要意义。本文提出一种解耦的3D分割流程,兼顾模块化与对新型3D表示及语义分割基础模型的适应性。首先通过3D高斯重建场景,并利用2D实例提案网络的对比监督学习无类别特征;随后对这些3D特征聚类生成粗粒度对象或部件级掩码;最后将每个3D聚类与2D开放词汇分割模型预测的类别感知掩码匹配,赋予语义标签而无需重训练3D表示。该解耦设计(1)提供即插即用接口以更换不同2D或3D模块,(2)无需额外开销即可实现多物体实例分割,(3)利用丰富3D几何信息提升场景理解鲁棒性。在合成与真实室内数据集上的评估表明,相比同类基于NeRF的管道,本方法在mIoU和mAcc上均有提升,尤其在挑战性或长尾类别上表现更优。我们还展示了不同2D主干网络对最终分割结果的影响,凸显框架的模块化优势。结果证实,解耦3D掩码提议与语义分类可实现灵活、高效且支持开放词汇的3D分割。
原文摘要 · Abstract (English)
Open-set 3D segmentation represents a major point of interest for multiple downstream robotics and augmented/virtual reality applications. We present a decoupled 3D segmentation pipeline to ensure modularity and adaptability to novel 3D representations as well as semantic segmentation foundation models. We first reconstruct a scene with 3D Gaussians and learn class-agnostic features through contrastive supervision from a 2D instance proposal network. These 3D features are then clustered to form coarse object- or part-level masks. Finally, we match each 3D cluster to class-aware masks predicted by a 2D open-vocabulary segmentation model, assigning semantic labels without retraining the 3D representation. Our decoupled design (1) provides a plug-and-play interface for swapping different 2D or 3D modules, (2) ensures multi-object instance segmentation at no extra cost, and (3) leverages rich 3D geometry for robust scene understanding. We evaluate on synthetic and real-world indoor datasets, demonstrating improved performance over comparable NeRF-based pipelines on mIoU and mAcc, particularly for challenging or long-tail classes. We also show how varying the 2D backbone affects the final segmentation, highlighting the modularity of our framework. These results confirm that decoupling 3D mask proposal and semantic classification can deliver flexible, efficient, and open-vocabulary 3D segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。