用2D模型生成的分割图增强稀疏3D标注,提升弱监督点云分割性能。
Integrating SAM Supervision for 3D Weakly Supervised Point Cloud Segmentation
- 利用2D基础模型生成分割掩码,通过几何对应传播至3D空间。
- 通过置信度与不确定性正则化,筛选可靠伪标签并扩展标注范围。
- 适合缺乏充足3D标注但有2D图像数据的研究者使用。
当前3D语义分割方法在标注有限的情况下训练模型,以应对大规模、不规则且无序的3D点云数据标注难题。这些方法通常仅关注3D域,未充分利用2D与3D数据的互补性。部分方法通过扩展原始标签或生成伪标签指导训练,但难以充分使用标签或处理其中噪声。随着2D基础模型的发展,我们提出一种新方法:通过引入2D基础模型生成的分割掩码,最大化稀疏3D标注的利用率。通过建立3D场景与2D视图之间的几何对应关系,将2D分割掩码传播至3D空间,并扩展稀疏标注覆盖区域,显著扩充可用标签池。进一步对3D点云的增强版本应用基于置信度和不确定性的一致性正则化,筛选可靠伪标签,并将其传播至3D掩码以生成更多标签。该策略有效弥合了有限3D标注与2D基础模型强大能力之间的差距,显著提升3D弱监督分割性能。
原文摘要 · Abstract (English)
Current methods for 3D semantic segmentation propose training models with limited annotations to address the difficulty of annotating large, irregular, and unordered 3D point cloud data. They usually focus on the 3D domain only, without leveraging the complementary nature of 2D and 3D data. Besides, some methods extend original labels or generate pseudo labels to guide the training, but they often fail to fully use these labels or address the noise within them. Meanwhile, the emergence of comprehensive and adaptable foundation models has offered effective solutions for segmenting 2D data. Leveraging this advancement, we present a novel approach that maximizes the utility of sparsely available 3D annotations by incorporating segmentation masks generated by 2D foundation models. We further propagate the 2D segmentation masks into the 3D space by establishing geometric correspondences between 3D scenes and 2D views. We extend the highly sparse annotations to encompass the areas delineated by 3D masks, thereby substantially augmenting the pool of available labels. Furthermore, we apply confidence- and uncertainty-based consistency regularization on augmentations of the 3D point cloud and select the reliable pseudo labels, which are further spread on the 3D masks to generate more labels. This innovative strategy bridges the gap between limited 3D annotations and the powerful capabilities of 2D foundation models, ultimately improving the performance of 3D weakly supervised segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。