根据深度变化动态融合激光雷达与图像特征，提升3D目标检测精度与鲁棒性。
DepthFusion: Depth-Aware Hybrid Feature Fusion for LiDAR-Camera 3D Object Detection
- 按深度信息分层调整图像与点云特征权重，实现自适应多模态融合。
- 在nuScenes和KITTI上性能超越现有方法，尤其在含噪声数据上表现更优。
- 适合需要高鲁棒性的自动驾驶感知系统，尤其应对复杂光照与遮挡场景。
当前先进的激光雷达-相机3D目标检测器主要关注特征融合策略，但忽略了深度对模态作用的影响。本文通过统计分析与可视化首次发现，不同模态在不同深度下发挥的作用不同。基于此，提出深度感知的混合特征融合（DepthFusion）策略，通过全局与局部深度编码引导点云与图像特征的权重分配。其中，全局融合模块（Depth-GFusion）利用深度编码自适应调整图像鸟瞰图（BEV）特征在多模态全局特征中的权重；局部融合模块（Depth-LFusion）则通过深度编码调节原始体素特征与多视角图像特征在局部特征中的权重，以弥补原始特征转为BEV空间时的信息损失。在nuScenes与KITTI数据集上的大量实验表明，DepthFusion优于此前最先进方法，且在nuScenes-C数据集上对各类噪声干扰更具鲁棒性。
原文摘要 · Abstract (English)
State-of-the-art LiDAR-camera 3D object detectors usually focus on feature fusion. However, they neglect the factor of depth while designing the fusion strategy. In this work, we are the first to observe that different modalities play different roles as depth varies via statistical analysis and visualization. Based on this finding, we propose a Depth-Aware Hybrid Feature Fusion (DepthFusion) strategy that guides the weights of point cloud and RGB image modalities by introducing depth encoding at both global and local levels. Specifically, the Depth-GFusion module adaptively adjusts the weights of image Bird's-Eye-View (BEV) features in multi-modal global features via depth encoding. Furthermore, to compensate for the information lost when transferring raw features to the BEV space, we propose a Depth-LFusion module, which adaptively adjusts the weights of original voxel features and multi-view image features in multi-modal local features via depth encoding. Extensive experiments on the nuScenes and KITTI datasets demonstrate that our DepthFusion method surpasses previous state-of-the-art methods. Moreover, our DepthFusion is more robust to various kinds of corruptions, outperforming previous methods on the nuScenes-C dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。