用3D结构引导多视角掩码匹配,实现无需标注的3D实例分割
MV3DIS: Multi-View Mask Matching via 3D Guides for Zero-Shot 3D Instance Segmentation
- 通过3D粗略分段作为跨视角匹配基准,提升2D掩码一致性
- 在多个数据集上优于现有方法,显著减少分割碎片化
- 适合需要零样本3D分割能力的研究者和工业应用
传统3D实例分割依赖人工标注的3D数据进行监督训练,限制了其可扩展性和对新物体的泛化能力。近期方法利用来自分割任意模型(SAM)的多视角2D掩码来指导3D几何原型的合并,从而实现零样本3D实例分割。然而,这些方法通常独立处理每一帧,仅依靠SAM预测分数等2D度量生成分割图,忽略了多视角关联与内在3D先验,导致跨视图2D掩码不一致,最终造成3D分割碎片化。本文提出MV3DIS,一种自粗到精的零样本3D实例分割框架,显式引入3D先验。具体而言,我们设计了3D引导的掩码匹配策略,以粗略3D片段为共同参考,跨视角匹配2D掩码,并通过3D覆盖分布强化多视角一致性。在这些视图一致的2D掩码引导下,粗略3D片段被进一步细化为精确的3D实例。此外,我们提出深度一致性加权机制,量化投影可靠性,抑制因物体间遮挡带来的歧义,提升3D到2D对应关系的鲁棒性。在ScanNetV2、ScanNet200、ScanNet++、Replica和Matterport3D等多个数据集上的大量实验表明,MV3DIS性能显著优于先前方法。
原文摘要 · Abstract (English)
Conventional 3D instance segmentation methods rely on labor-intensive 3D annotations for supervised training, which limits their scalability and generalization to novel objects. Recent approaches leverage multi-view 2D masks from the Segment Anything Model (SAM) to guide the merging of 3D geometric primitives, thereby enabling zero-shot 3D instance segmentation. However, these methods typically process each frame independently and rely solely on 2D metrics, such as SAM prediction scores, to produce segmentation maps. This design overlooks multi-view correlations and inherent 3D priors, leading to inconsistent 2D masks across views and ultimately fragmented 3D segmentation. In this paper, we propose MV3DIS, a coarse-to-fine framework for zero-shot 3D instance segmentation that explicitly incorporates 3D priors. Specifically, we introduce a 3D-guided mask matching strategy that uses coarse 3D segments as a common reference to match 2D masks across views and consolidates multi-view mask consistency via 3D coverage distributions. Guided by these view-consistent 2D masks, the coarse 3D segments are further refined into precise 3D instances. Additionally, we introduce a depth consistency weighting scheme that quantifies projection reliability to suppress ambiguities from inter-object occlusions, thereby improving the robustness of 3D-to-2D correspondence. Extensive experiments on the ScanNetV2, ScanNet200, ScanNet++, Replica, and Matterport3D datasets demonstrate the effectiveness of MV3DIS, which achieves superior performance over previous methods
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。