用高斯点云先分对象再重建,实现3D开放词汇分割。
Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting
- 先分后重建:将高斯点云按对象分离,再进行3D重建
- 支持静态与动态场景,避免多视角不一致问题
- 结合CLIP实现开放词汇查询,适合机器人与AR应用
3D空间中的开放词汇查询对机器人、自动驾驶和增强现实等应用至关重要。然而,现有方法多依赖2D像素级分割,导致多视角不一致且3D物体检索效果差。同时,这些方法仅限于静态场景,难以处理动态场景,因运动建模复杂。本文提出Segment then Splat,一种基于高斯点云的3D感知开放词汇分割方法,适用于静态与动态场景。该方法反向传统“重建后分割”流程,先将高斯点云划分为不同对象集合,再完成重建。重建完成后,场景自然被分割为独立对象,实现真正3D分割。该设计消除了几何与语义模糊性及动态场景中高斯-对象错位问题,并加速优化过程,无需学习额外语言场。优化后,每个对象分配一个CLIP嵌入,支持开放词汇查询。在多个数据集上的实验验证了该方法在静态与动态场景下的有效性。
原文摘要 · Abstract (English)
Open-vocabulary querying in 3D space is crucial for enabling more intelligent perception in applications such as robotics, autonomous systems, and augmented reality. However, most existing methods rely on 2D pixel-level parsing, leading to multi-view inconsistencies and poor 3D object retrieval. Moreover, they are limited to static scenes and struggle with dynamic scenes due to the complexities of motion modeling. In this paper, we propose Segment then Splat, a 3D-aware open vocabulary segmentation approach for both static and dynamic scenes based on Gaussian Splatting. Segment then Splat reverses the long established approach of "segmentation after reconstruction" by dividing Gaussians into distinct object sets before reconstruction. Once reconstruction is complete, the scene is naturally segmented into individual objects, achieving true 3D segmentation. This design eliminates both geometric and semantic ambiguities, as well as Gaussian-object misalignment issues in dynamic scenes. It also accelerates the optimization process, as it eliminates the need for learning a separate language field. After optimization, a CLIP embedding is assigned to each object to enable open-vocabulary querying. Extensive experiments one various datasets demonstrate the effectiveness of our proposed method in both static and dynamic scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。