arXiv:2410.17505cs.CV2024-10被引 24

用3D高斯点云实现快速且鲁棒的全景分割,抗噪声能力强。

PLGS: Robust Panoptic Lifting with 3D Gaussian Splatting

  • 构建结构化3D高斯模型引入平滑性,提升分割一致性。
  • 在多个基准上优于现有方法,速度远超基于NeRF的方法。
  • 适合需要高效高精度3D全景分割的应用场景。

以往方法使用神经辐射场(NeRF)进行全景提升,但训练与渲染速度不理想。相比之下,3D高斯点云(3DGS)因其快速训练与渲染成为主流技术。然而,传统3DGS缺乏参数化结构支持,难以满足基本平滑性假设,对2D分割掩码中的噪声更敏感。本文提出PLGS,使3DGS在保持高效的同时,能从噪声2D掩码生成一致的全景分割结果。我们构建了面向全景的结构化3D高斯模型,并设计有效降噪策略;语义场不再依赖结构光重建初始化,而是通过可靠语义锚点引导3D高斯初始化,并作为训练中的平滑正则;同时引入自训练机制,利用融合渲染掩码与噪声掩码生成伪标签以增强鲁棒性。实例场将2D实例掩码投影至3D空间,匹配方向包围盒生成跨视角一致的实例掩码用于监督。大量实验表明,本方法在分割质量与速度上均超越现有最优方法。

原文摘要 · Abstract (English)

Previous methods utilize the Neural Radiance Field (NeRF) for panoptic lifting, while their training and rendering speed are unsatisfactory. In contrast, 3D Gaussian Splatting (3DGS) has emerged as a prominent technique due to its rapid training and rendering speed. However, unlike NeRF, the conventional 3DGS may not satisfy the basic smoothness assumption as it does not rely on any parameterized structures to render (e.g., MLPs). Consequently, the conventional 3DGS is, in nature, more susceptible to noisy 2D mask supervision. In this paper, we propose a new method called PLGS that enables 3DGS to generate consistent panoptic segmentation masks from noisy 2D segmentation masks while maintaining superior efficiency compared to NeRF-based methods. Specifically, we build a panoptic-aware structured 3D Gaussian model to introduce smoothness and design effective noise reduction strategies. For the semantic field, instead of initialization with structure from motion, we construct reliable semantic anchor points to initialize the 3D Gaussians. We then use these anchor points as smooth regularization during training. Additionally, we present a self-training approach using pseudo labels generated by merging the rendered masks with the noisy masks to enhance the robustness of PLGS. For the instance field, we project the 2D instance masks into 3D space and match them with oriented bounding boxes to generate cross-view consistent instance masks for supervision. Experiments on various benchmarks demonstrate that our method outperforms previous state-of-the-art methods in terms of both segmentation quality and speed.

3D高斯全景分割鲁棒性高效建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。