仅用NeRF实现开放词汇3D全景分割,保持实例一致性与唯一性。
Cues3D: Unleashing the Power of Sole NeRF for Consistent and Unique Instances in Open-Vocabulary 3D Panoptic Segmentation

- 基于NeRF隐式三维场构建全局一致几何,无需跨视角预对齐。
- 提出三阶段训练框架,通过初始知识修正实例ID,确保唯一性。
- 在多个数据集上优于纯2D方法,融合点云时性能超越主流3D-2D融合模型。
开放词汇3D全景分割成为新趋势,现有高性能方法依赖2D分割与几何感知3D原型结合,但需高保真3D点云支持,如基于神经辐射场(NeRF)的方法。然而,此类方法在部分观测下难以维持一致性。现有工作采用对比损失或跨视图关联预处理来增强视图一致性,而本文提出Cues3D,一种仅依赖NeRF的紧凑方案。核心思想是:NeRF的隐式3D场天然具备全局一致性,可无需显式跨视图监督实现有效物体区分。我们设计三阶段训练流程——初始化-消歧-优化,利用初始学习知识修正实例ID;并提出实例消歧方法,匹配NeRF渲染的3D掩码,确保全局唯一的3D实例身份。实验在ScanNet v2、ScanNet200、ScanNet++和Replica数据集上进行,涵盖3D实例、全景与语义分割任务。Cues3D在不依赖额外点云时优于其他2D图像基方法,使用额外3D点云时甚至超越最新2D-3D融合方法。代码将发布于github。
原文摘要 · Abstract (English)
Open-vocabulary 3D panoptic segmentation has recently emerged as a significant trend. Top-performing methods currently integrate 2D segmentation with geometry-aware 3D primitives. However, the advantage would be lost without high-fidelity 3D point clouds, such as methods based on Neural Radiance Field (NeRF). These methods are limited by the insufficient capacity to maintain consistency across partial observations. To address this, recent works have utilized contrastive loss or cross-view association pre-processing for view consensus. In contrast to them, we present Cues3D, a compact approach that relies solely on NeRF instead of pre-associations. The core idea is that NeRF's implicit 3D field inherently establishes a globally consistent geometry, enabling effective object distinction without explicit cross-view supervision. We propose a three-phase training framework for NeRF, initialization-disambiguation-refinement, whereby the instance IDs are corrected using the initially-learned knowledge. Additionally, an instance disambiguation method is proposed to match NeRF-rendered 3D masks and ensure globally unique 3D instance identities. With the aid of Cues3D, we obtain highly consistent and unique 3D instance ID for each object across views with a balanced version of NeRF. Our experiments are conducted on ScanNet v2, ScanNet200, ScanNet++, and Replica datasets for 3D instance, panoptic, and semantic segmentation tasks. Cues3D outperforms other 2D image-based methods and competes with the latest 2D-3D merging based methods, while even surpassing them when using additional 3D point clouds. The code link could be found in the appendix and will be released on \href{https://github.com/mRobotit/Cues3D}{github}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。