用大模型生成的鸟瞰图地图，提升自动驾驶3D目标检测与跟踪性能
Bridging Perspectives: Foundation Model Guided BEV Maps for 3D Object Detection and Tracking
- 融合透视图与鸟瞰图特征，通过大模型蒸馏构建语义几何增强的鸟瞰图
- 在nuScenes和Argoverse 2上实现当前最佳性能，3D检测精度显著提升
- 适合关注自动驾驶感知、多视角融合与基础模型应用的研究者
基于摄像头的3D目标检测与跟踪对自动驾驶感知至关重要。现有先进方法通常仅依赖透视图（PV）或鸟瞰图（BEV）特征，难以同时利用细粒度物体细节与结构化空间表征。本文提出DualViewDistill，一种结合PV与BEV图像特征的混合检测与跟踪框架，通过基础模型引导生成的BEV地图，将描述性DINOv2特征经新颖蒸馏过程融入BEV表示。通过可变形聚合机制整合PV特征与富含语义和几何信息的BEV地图，显著提升3D目标检测与跟踪性能。在nuScenes与Argoverse 2基准上的大量实验表明，DualViewDistill达到当前最优表现，验证了基础模型生成的BEV地图在自动驾驶可靠感知中的潜力。代码与预训练模型已公开于https://dualviewdistill.cs.uni-freiburg.de。
原文摘要 · Abstract (English)
Camera-based 3D object detection and tracking are essential for perception in autonomous driving. Current state-of-the-art approaches often rely exclusively on either perspective-view (PV) or bird's-eye-view (BEV) features, limiting their ability to leverage both fine-grained object details and spatially structured scene representations. In this work, we propose DualViewDistill, a hybrid detection and tracking framework that incorporates both PV and BEV camera image features to leverage their complementary strengths. Our approach introduces BEV maps guided by foundation models, leveraging descriptive DINOv2 features that are distilled into BEV representations through a novel distillation process. By integrating PV features with BEV maps enriched with semantic and geometric features from DINOv2, our model leverages this hybrid representation via deformable aggregation to enhance 3D object detection and tracking. Extensive experiments on the nuScenes and Argoverse 2 benchmarks demonstrate that DualViewDistill achieves state-of-the-art performance. The results showcase the potential of foundation model BEV maps to enable more reliable perception for autonomous driving. We make the code and pre-trained models available at https://dualviewdistill.cs.uni-freiburg.de .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。