arXiv:2503.15285cs.CV2025-03

将点云投影到2D图匹配图像,实现高精度实时跨模态配准。

EEPNet-V2: Patch-to-Pixel Solution for Efficient Cross-Modal Registration between LiDAR Point Cloud and Camera Image

  • 点云转2D图,用多尺度网络提取图像与点云投影特征。
  • 在KITTI上达到99%以上配准准确率,支持实时运行。
  • 无需外部标定板,适合自动驾驶等实时系统应用。

跨模态数据融合的关键是精确对齐不同传感器的数据。然而,激光雷达点云与相机图像之间的标定通常耗时且需外部标定板或特定环境特征。跨模态配准通过直接对齐数据解决此问题,无需外部标定。但由于点云与图像间存在领域差异,现有方法难以在保持实时性能的同时达到满意配准精度。为此,我们提出一种框架,将点云投影为多个2D表示,用于与相机图像匹配,既有效利用激光雷达点云的几何特性,又缩小了点云与图像间的领域差距。此外,为应对跨模态差异及点云与图像在图像匹配任务中重叠度低的问题,我们引入多尺度特征提取网络,有效提取相机图像与点云投影图的特征。同时,提出一种像素级补丁匹配网络,提供更有效的监督并实现高精度配准。我们在KITTI和nuScenes数据集上验证了模型性能。实验结果表明,所提方法具备实时性能和极高的配准精度:在KITTI数据集上,配准准确率超过99%。代码已开源:https://github.com/ESRSchao/EEPNet-V2。

原文摘要 · Abstract (English)

The primary requirement for cross-modal data fusion is the precise alignment of data from different sensors. However, the calibration between LiDAR point clouds and camera images is typically time-consuming and needs external calibration board or specific environmental features. Cross-modal registration effectively solves this problem by aligning the data directly without requiring external calibration. However, due to the domain gap between the point cloud and the image, existing methods rarely achieve satisfactory registration accuracy while maintaining real-time performance. To address this issue, we propose a framework that projects point clouds into several 2D representations for matching with camera images, which not only leverages the geometric characteristic of LiDAR point clouds effectively but also bridge the domain gap between the point cloud and image. Moreover, to tackle the challenges of cross modal differences and the limited overlap between LiDAR point clouds and images in the image matching task, we introduce a multi-scale feature extraction network to effectively extract features from both camera images and the projection maps of LiDAR point cloud. Additionally, we propose a patch-to-pixel matching network to provide more effective supervision and achieve high accuracy. We validate the performance of our model through experiments on the KITTI and nuScenes datasets. Experimental results demonstrate the the proposed method achieves real-time performance and extremely high registration accuracy. Specifically, on the KITTI dataset, our model achieves a registration accuracy rate of over 99\%. Our code is released at: https://github.com/ESRSchao/EEPNet-V2.

跨模态配准激光雷达实时系统点云图像融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。