融合雷达与激光雷达,提升恶劣天气下3D目标检测性能。
V2X-R: Cooperative LiDAR-4D Radar Fusion with Denoising Diffusion for 3D Object Detection
- 设计多模态去噪扩散模块,用雷达特征指导修复受损激光雷达数据。
- 在12,079个场景的V2X-R数据集上,雾天/雪天性能提升5.73%/6.70%。
- 首个含激光雷达、摄像头和4D雷达的模拟车联网数据集,适合自动驾驶研究者。
当前车联网(V2X)系统通过融合激光雷达与相机数据显著提升了3D目标检测性能,但在恶劣天气下表现下降。具备多普勒信息与鲁棒几何特征的4D雷达为解决该问题提供了可能。为此,本文提出V2X-R,首个融合激光雷达、相机与4D雷达的仿真车联网数据集,包含12,079个场景、37,727帧激光雷达与4D雷达点云、150,908张图像及170,859个标注的3D车辆边界框。进一步,我们设计了一种新型协同激光雷达-4D雷达融合检测流程,并引入多模态去噪扩散(MDD)模块以增强天气鲁棒性。MDD利用稳健的4D雷达特征作为条件,引导扩散模型对噪声激光雷达特征进行去噪。实验表明,所提融合方法在V2X-R上表现优异;此外,MDD模块使基础融合模型在雾天/雪天条件下性能提升最高达5.73%/6.70%,且对正常条件影响极小。数据集与代码将公开于:https://github.com/ylwhxht/V2X-R。
原文摘要 · Abstract (English)
Current Vehicle-to-Everything (V2X) systems have significantly enhanced 3D object detection using LiDAR and camera data. However, these methods suffer from performance degradation in adverse weather conditions. The weather-robust 4D radar provides Doppler and additional geometric information, raising the possibility of addressing this challenge. To this end, we present V2X-R, the first simulated V2X dataset incorporating LiDAR, camera, and 4D radar. V2X-R contains 12,079 scenarios with 37,727 frames of LiDAR and 4D radar point clouds, 150,908 images, and 170,859 annotated 3D vehicle bounding boxes. Subsequently, we propose a novel cooperative LiDAR-4D radar fusion pipeline for 3D object detection and implement it with various fusion strategies. To achieve weather-robust detection, we additionally propose a Multi-modal Denoising Diffusion (MDD) module in our fusion pipeline. MDD utilizes weather-robust 4D radar feature as a condition to prompt the diffusion model to denoise noisy LiDAR features. Experiments show that our LiDAR-4D radar fusion pipeline demonstrates superior performance in the V2X-R dataset. Over and above this, our MDD module further improved the performance of basic fusion model by up to 5.73%/6.70% in foggy/snowy conditions with barely disrupting normal performance. The dataset and code will be publicly available at: https://github.com/ylwhxht/V2X-R.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。