arXiv:2409.13325cs.CV2024-09

用少量标注数据实现点云与图像联合分割,提升标注效率。

Towards Semi-supervised Dual-modal Semantic Segmentation

  • 设计双流并行网络,融合少量标注点云、大量无标注点云和图像。
  • 通过伪标签优化与自训练机制,在两个公开数据集上性能超越多数半监督方法。
  • 适合需要降低人工标注成本的三维场景理解任务,如自动驾驶感知。

随着3D与2D数据采集技术的发展,同时获取场景的点云与图像变得容易,进一步推动了双模态语义分割的发展。现有大多数点云与图像联合分割方法严重依赖标注数据的数量与质量,而点级与像素级标注耗时且人力密集。为解决此问题,我们提出一种并行双流网络PD-Net,用于处理半监督双模态语义分割任务,该方法联合利用少量标注点云、大量无标注点云及无标注图像。PD-Net包含两条并行分支:原始流与伪标签预测流。后者对无标注点云及其对应图像生成伪标签,随后将这些数据送入原始流进行自训练。每条流均包含针对3D与2D数据的编码器-解码器分支,并引入多个双模态融合模块以融合多模态特征。此外,还设计了伪标签优化模块以改进伪标签质量。在两个公开数据集上的实验结果表明,所提方法不仅优于多种半监督对比方法,且在多数情况下达到与部分全监督方法相当的性能。

原文摘要 · Abstract (English)

With the development of 3D and 2D data acquisition techniques, it has become easy to obtain point clouds and images of scenes simultaneously, which further facilitates dual-modal semantic segmentation. Most existing methods for simultaneously segmenting point clouds and images rely heavily on the quantity and quality of the labeled training data. However, massive point-wise and pixel-wise labeling procedures are time-consuming and labor-intensive. To address this issue, we propose a parallel dual-stream network to handle the semi-supervised dual-modal semantic segmentation task, called PD-Net, by jointly utilizing a small number of labeled point clouds, a large number of unlabeled point clouds, and unlabeled images. The proposed PD-Net consists of two parallel streams (called original stream and pseudo-label prediction stream). The pseudo-label prediction stream predicts the pseudo labels of unlabeled point clouds and their corresponding images. Then, the unlabeled data is sent to the original stream for self-training. Each stream contains two encoder-decoder branches for 3D and 2D data respectively. In each stream, multiple dual-modal fusion modules are explored for fusing the dual-modal features. In addition, a pseudo-label optimization module is explored to optimize the pseudo labels output by the pseudo-label prediction stream. Experimental results on two public datasets demonstrate that the proposed PD-Net not only outperforms the comparative semi-supervised methods but also achieves competitive performances with some fully-supervised methods in most cases.

双模态分割半监督学习点云处理自训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。