融合点云与图像，用标定硬关联提升3D检测精度
DecoratingFusion: A LiDAR-Camera Fusion Network with the Combination of Point-level and Feature-level Fusion
- 结合点级与特征级融合，利用标定矩阵建立硬关联
- 在KITTI和Waymo上显著优于现有方法，提升检测性能
- 适合需要高精度3D感知的自动驾驶场景
激光雷达与摄像头在自动驾驶中提供互补的3D检测信息。当前主流融合方法多在特征层进行，依赖学习到的点云与图像间软关联，缺乏可解释性且忽略由标定矩阵决定的硬关联。本文提出DecoratingFusion，将点级与特征级融合结合，利用标定矩阵建立的硬关联引导目标查询生成。在早期融合阶段，使用图像2D CNN特征对点云数据进行装饰，并通过两个独立稀疏卷积提取装饰后点云特征。在中层融合阶段，以中心热图初始化查询，并将预测类别标签作为辅助信息嵌入查询，使初始位置更接近真实目标中心。在KITTI和Waymo两个主流数据集上的大量实验表明，DecoratingFusion性能显著领先。
原文摘要 · Abstract (English)
Lidars and cameras play essential roles in autonomous driving, offering complementary information for 3D detection. The state-of-the-art fusion methods integrate them at the feature level, but they mostly rely on the learned soft association between point clouds and images, which lacks interpretability and neglects the hard association between them. In this paper, we combine feature-level fusion with point-level fusion, using hard association established by the calibration matrices to guide the generation of object queries. Specifically, in the early fusion stage, we use the 2D CNN features of images to decorate the point cloud data, and employ two independent sparse convolutions to extract the decorated point cloud features. In the mid-level fusion stage, we initialize the queries with a center heatmap and embed the predicted class labels as auxiliary information into the queries, making the initial positions closer to the actual centers of the targets. Extensive experiments conducted on two popular datasets, i.e. KITTI, Waymo, demonstrate the superiority of DecoratingFusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。