arXiv:2411.03819cs.CV2024-11NeurIPS被引 3

利用3D先验提升任意3D实例分割精度,解决几何相似物误分问题。

SA3DIP: Segment Any 3D Instance with Potential 3D Priors

论文配图:SA3DIP: Segment Any 3D Instance with Potential 3D Priors
图 1 · 摘自论文原文
  • 融合几何与纹理先验生成互补3D基本单元,减少初始误差。
  • 引入3D检测器提供空间约束,优化实例合并过程,缓解过分割。
  • 构建新标注数据集ScanNetV2-INS,支持更公平的无类别实例分割评估。

2D基础模型的兴起推动了开放世界3D实例分割的研究。现有方法以超点为几何单元,结合Segment Anything模型(SAM)的多视图掩码进行合并引导,实现出色的零样本分割效果。但其对3D先验利用有限,仅基于空间坐标估计法向量计算超点,导致几何相似实例出现欠分割。同时,过度依赖SAM及手工算法在2D空间易引发过分割,因SAM本身倾向于部件级分割。为此,本文提出SA3DIP,通过融合几何与纹理先验生成互补3D基本单元,降低后续流程误差;并引入3D检测器提供的空间约束,指导进一步合并。此外,发现ScanNetV2基准存在大量低质量标注,影响评估公平性,因此构建了包含完整标签的ScanNetV2-INS数据集,并补充实例用于3D类无关实例分割。在多个2D-3D数据集上的实验验证了方法的有效性与鲁棒性。代码与数据集已公开。

原文摘要 · Abstract (English)

The proliferation of 2D foundation models has sparked research into adapting them for open-world 3D instance segmentation. Recent methods introduce a paradigm that leverages superpoints as geometric primitives and incorporates 2D multi-view masks from Segment Anything model (SAM) as merging guidance, achieving outstanding zero-shot instance segmentation results. However, the limited use of 3D priors restricts the segmentation performance. Previous methods calculate the 3D superpoints solely based on estimated normal from spatial coordinates, resulting in under-segmentation for instances with similar geometry. Besides, the heavy reliance on SAM and hand-crafted algorithms in 2D space suffers from over-segmentation due to SAM's inherent part-level segmentation tendency. To address these issues, we propose SA3DIP, a novel method for Segmenting Any 3D Instances via exploiting potential 3D Priors. Specifically, on one hand, we generate complementary 3D primitives based on both geometric and textural priors, which reduces the initial errors that accumulate in subsequent procedures. On the other hand, we introduce supplemental constraints from the 3D space by using a 3D detector to guide a further merging process. Furthermore, we notice a considerable portion of low-quality ground truth annotations in ScanNetV2 benchmark, which affect the fair evaluations. Thus, we present ScanNetV2-INS with complete ground truth labels and supplement additional instances for 3D class-agnostic instance segmentation. Experimental evaluations on various 2D-3D datasets demonstrate the effectiveness and robustness of our approach. Our code and proposed ScanNetV2-INS dataset are available HERE.

3D分割实例分割先验知识数据集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。