arXiv:2507.16861cs.CVcs.AI2025-07中稿 · CVPR被引 3

用2D检测结果预对齐激光雷达与摄像头特征,提升3D目标检测精度。

Look Before You Fuse: 2D-Guided Cross-Modal Alignment for Robust 3D Detection

  • 利用2D物体边界先验预测并修正跨模态特征错位
  • 在nuScenes上达到71.5% mAP、73.6% NDS,SOTA表现
  • 适合自动驾驶中多传感器融合的鲁棒3D检测任务

将激光雷达与摄像头数据融合至统一的鸟瞰图(BEV)表示,对提升自动驾驶的3D感知能力至关重要。然而,现有方法存在激光雷达与摄像头特征的空间错位问题,导致相机分支深度监督不准确,跨模态特征融合出错。根源在于投影误差,由标定不准和滚动快门效应引起。本文关键洞察是:此类误差位置并非随机,而是集中在物体-背景边界处,2D检测器可可靠识别。基于此,提出利用2D物体先验在融合前预对齐跨模态特征。针对局部错位,设计先验引导深度校准(PGDC),缓解错位并保留正确特征对;针对全局错位,引入不连续感知几何融合(DAGF),抑制PGDC残余噪声,并显式增强物体-背景边界的深度突变,生成结构感知表示。为有效利用对齐特征,引入结构引导深度调制器(SGDM),采用门控注意力机制高效融合对齐后的深度与图像特征。在nuScenes验证集上,mAP达71.5%,NDS达73.6%;在Argoverse 2验证集上,mAP达41.7%,性能领先。

原文摘要 · Abstract (English)

Integrating LiDAR and camera inputs into a unified Bird's-Eye-View (BEV) representation is crucial for enhancing 3D perception capabilities of autonomous vehicles. However, existing methods suffer from spatial misalignment between LiDAR and camera features, which causes inaccurate depth supervision in camera branch and erroneous fusion during cross-modal feature aggregation. The root cause of this misalignment lies in projection errors, stemming from calibration inaccuracies and rolling shutter effect. The key insight of this work is that locations of these projection errors are not random but highly predictable, as they are concentrated at object-background boundaries which 2D detectors can reliably identify. Based on this, our main motivation is to utilize 2D object priors to pre-align cross-modal features before fusion. To address local misalignment, we propose Prior Guided Depth Calibration (PGDC), which leverages 2D priors to alleviate misalignment and preserve correct cross-modal feature pairs. To resolve global misalignment, we introduce Discontinuity Aware Geometric Fusion (DAGF) to suppress residual noise from PGDC and explicitly enhance sharp depth transitions at object-background boundaries, yielding a structurally aware representation. To effectively utilize these aligned representations, we incorporate Structural Guidance Depth Modulator (SGDM), using a gated attention mechanism to efficiently fuse aligned depth and image features. Our method achieves SOTA performance on nuScenes validation dataset, with its mAP and NDS reaching 71.5% and 73.6% respectively. Additionally, on the Argoverse 2 validation set, we achieve a competitive mAP of 41.7%.

3D检测多模态融合自动驾驶特征对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。