用雷达点云生成伪掩码，结合扩散模型补全图像，提升恶劣天气下的分割精度
CaRaFFusion: Improving 2D Semantic Segmentation with Camera-Radar Point Cloud Fusion and Zero-Shot Image Inpainting
- 以雷达点为提示生成伪掩码，融合视觉与雷达信息
- 在Waterscenes数据集上提升相机基线mIoU 2.63%
- 适合自动驾驶中多模态感知与低光照/雨雾场景应用
环境语义分割对自动驾驶和机器人至关重要，有助于理解周围场景。虽然摄像头提供丰富的视觉细节，但在恶劣天气下表现不佳；而雷达虽鲁棒性强，但数据稀疏且噪声大。为此，本文提出一种新框架，在相机-雷达融合架构中引入扩散模型，利用雷达点特征通过Segment-Anything模型生成伪掩码（将投影雷达点作为点提示），并设计去噪单元优化伪掩码质量，再用于生成补全图像以恢复原始图像缺失信息。该方法在相机仅基线上实现mIoU提升2.63%，在相机-雷达融合架构上提升1.48%（在Waterscenes数据集上）。结果表明，该方法在恶劣天气条件下有效提升语义分割性能。
原文摘要 · Abstract (English)
Segmenting objects in an environment is a crucial task for autonomous driving and robotics, as it enables a better understanding of the surroundings of each agent. Although camera sensors provide rich visual details, they are vulnerable to adverse weather conditions. In contrast, radar sensors remain robust under such conditions, but often produce sparse and noisy data. Therefore, a promising approach is to fuse information from both sensors. In this work, we propose a novel framework to enhance camera-only baselines by integrating a diffusion model into a camera-radar fusion architecture. We leverage radar point features to create pseudo-masks using the Segment-Anything model, treating the projected radar points as point prompts. Additionally, we propose a noise reduction unit to denoise these pseudo-masks, which are further used to generate inpainted images that complete the missing information in the original images. Our method improves the camera-only segmentation baseline by 2.63% in mIoU and enhances our camera-radar fusion architecture by 1.48% in mIoU on the Waterscenes dataset. This demonstrates the effectiveness of our approach for semantic segmentation using camera-radar fusion under adverse weather conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。