arXiv:2607.12959cs.CV2026-07

用视觉大模型增强车载3D感知,让多车协作检测更准

ViCo3D: Empowering LiDAR-based Collaborative 3D Object Detection with Vision Foundation Models

论文配图:ViCo3D: Empowering LiDAR-based Collaborative 3D Object Detection with Vision Foundation Models
图 1 · 摘自论文原文
  • 将点云转为三通道图像,让DINOv2提取语义特征
  • 融合视觉与几何特征,在DAIR-V2X上实现1.8倍协作增益
  • 适合研究车联网协同感知的学者和工程师

基于激光雷达的车联万物(V2X)系统中,协作式3D感知通常依赖于跨智能体的鸟瞰图(BEV)特征融合。然而,现有BEV表示多由从零训练的激光雷达主干网络提取,以几何信息为主,缺乏通用语义先验,限制了特征级协作效果。相比之下,预训练于大规模图像数据的视觉基础模型(VFMs)在2D任务中展现出强大的通用视觉表征能力,具备提升单智能体激光雷达BEV表示的潜力。但由于图像与点云之间的模态差异,将其应用于激光雷达3D检测仍具挑战。为此,本文提出ViCo3D,一个由视觉基础模型驱动的协作式3D目标检测框架。具体而言,ViCo3D从三方面适配VFMs:首先,将点云投影至BEV平面生成三通道图像,使DINOv2可从激光雷达输入中提取BEV空间视觉特征;其次,设计多尺度BEV融合模块,有效整合DINOv2提取的视觉特征与原始激光雷达几何特征;此外,采用以我车为中心的跨智能体融合策略,聚合多智能体间的互补信息。在DAIR-V2X和V2XSet数据集上的实验表明,ViCo3D达到当前最优的3D检测性能,尤其在DAIR-V2X上相较先前方法实现高达1.8倍的协作增益。代码将公开以供后续研究。

原文摘要 · Abstract (English)

LiDAR-based collaborative 3D perception in Vehicle-to-Everything (V2X) systems typically relies on fusing bird's-eye-view (BEV) features across agents. However, current BEV representations, typically extracted by LiDAR backbones trained from scratch, are geometry-dominated and lack general semantic priors, inherently limiting the efficacy of feature-level collaboration. Meanwhile, vision foundation models (VFMs) pretrained on large-scale image data have demonstrated strong capability in learning general-purpose and informative visual representations for 2D tasks, and have the potential to enhance agent-wise LiDAR BEV representations for collaboration. Despite this potential, adapting VFMs to LiDAR-based 3D detection remains challenging due to the substantial image-point cloud modality gap. To bridge this gap, we propose ViCo3D, a collaborative 3D object detection framework powered by VFMs. Specifically, ViCo3D adapts VFMs to LiDAR-based collaborative perception from three aspects: First, ViCo3D projects point clouds onto the BEV plane as three-channel images, enabling DINOv2 to extract BEV-space visual features from LiDAR inputs. Besides, to effectively integrate these DINOv2-derived features with LiDAR geometric features, ViCo3D introduces a multi-scale BEV fusion module within the single-agent encoder. In addition, ViCo3D adopts an ego-centric cross-agent fusion strategy to aggregate complementary information from multiple agents. Experiments on DAIR-V2X and V2XSet demonstrate that ViCo3D achieves state-of-the-art 3D detection performance. Remarkably, it delivers up to 1.8x greater collaborative gains than prior methods on DAIR-V2X. The code will be made public available for future investigation.

3D检测视觉大模型车联网点云处理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。