arXiv:2609.09005cs.CV2026-09

用视觉大模型实现相机与激光雷达跨模态定位,无需专门编码器。

DXPR: Depth-Based Vision-LiDAR Cross-Modal Place Recognition Using Vision Foundation Models

论文配图:DXPR: Depth-Based Vision-LiDAR Cross-Modal Place Recognition Using Vision Foundation Models
图 1 · 摘自论文原文
  • 将图像和激光雷达转为统一深度图,用一个视觉大模型提取通用特征。
  • 在KITTI上召回率接近完美,在昼夜/季节变化下仍稳定表现。
  • 适合需要低成本相机定位的机器人和自动驾驶场景。

我们提出DXPR,一种基于深度的跨模态定位框架,利用视觉基础模型(VFMs)将单目相机查询与预先构建的激光雷达地图匹配,无需模态专用编码器。该方法使机器人和自动驾驶车辆仅通过相机即可在预建激光雷达地图中实现鲁棒定位,即使在严重季节、天气和光照变化下依然有效。核心思路是将相机图像和激光雷达扫描统一转换为深度图像表示,使单一视觉基础模型主干配合聚合头学习模态不变的全局描述符。为确保成对度量学习忠实于场景几何,我们引入几何感知重叠矿工:在相机与激光雷达深度进行跨模态尺度对齐后,通过前向变形在视图间计算像素级重叠分数,用于重新标注模糊样本,并自适应调节多相似性损失中的正样本边界,避免在弱重叠视图上过拟合。在KITTI里程计和Boreas数据集上的大量实验表明,该方法在季节、天气和昼夜变化下均表现出强性能与鲁棒性。在KITTI上,多数序列的Recall@1接近完美,优于现有跨模态定位基线。在Boreas上,内部序列性能与强单模态基线(DINOv2-SALAD)相当,而在更具挑战性的跨序列设置中表现更优。相较于RangeBEV,本方法在内部与跨序列评估中均持续领先,验证了其在多样季节与光照变化下的鲁棒性。

原文摘要 · Abstract (English)

We present DXPR, a depth-based cross-modal place recognition (CMPR) framework that uses vision foundation models (VFMs) to match monocular camera queries against a LiDAR map without modality-specific encoders. This enables robots and autonomous vehicles to robustly localize using only cameras within pre-built LiDAR maps, even under severe seasonal, weather, and illumination changes. The key idea is to convert both camera images and LiDAR scans into a unified depth image representation so that a single VFM backbone with an aggregation head can learn modality-invariant global descriptors. To make pairwise metric learning faithful to scene geometry, we introduce a geometry-aware overlap miner: after cross-modal scale alignment of camera and LiDAR depth, we forward-warp measurements between views to compute a pixel-level overlap score. This score relabels ambiguous pairs and adaptively modulates the positive margin in a multi-similarity loss to avoid overfitting on weakly overlapping views. Extensive experiments on KITTI odometry and Boreas demonstrate strong performance and robustness across seasons, weather, and day/night. On KITTI, DXPR achieves near-perfect Recall@1 on most sequences and outperforms prior CMPR baselines. On Boreas, DXPR achieves intra-sequence performance on par with a strong single-modal baseline (DINOv2-SALAD), while showing clear improvements in the more challenging inter-sequence setting. Compared with RangeBEV, our method consistently performs better in both intra- and inter-sequence evaluations, demonstrating robustness under diverse seasonal and illumination changes.

跨模态定位视觉大模型深度估计自动驾驶

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。