用冻结的视觉大模型，让鸟瞰感知模型少训练数据、性能更强。
Revisiting Birds Eye View Perception Models with Frozen Foundation Models: DINOv2 and Metric3Dv2
- 用DINOv2和Metric3Dv2提取特征与深度，替代传统训练方式。
- 仅用一半数据和迭代次数，提升7.4个IoU，超越基线。
- 将深度信息转为伪LiDAR点云，适配纯摄像头系统。
鸟瞰视图感知模型需大量数据才能有效泛化。尽管传统数据集提供多样驾驶场景，但并非总能获取充足数据。因此必须最大化现有数据利用率。随着DINOv2和Metric3Dv2等大基础模型的出现，一个关键问题浮现：能否将其融入现有架构，不仅减少训练数据需求，还能超越当前模型性能？本文选取车辆分割领域的两种架构进行改进：Lift-Splat-Shoot与Simple-BEV。在Lift-Splat-Shoot中，采用冻结的DINOv2进行特征提取，结合Metric3Dv2进行深度估计，在仅使用一半训练数据和迭代次数的条件下，性能提升7.4 IoU。此外，创新性地将Metric3Dv2的深度信息转化为伪LiDAR点云，嵌入Simple-BEV架构，取代传统激光雷达，相比纯摄像头模型实现+3 IoU提升。
原文摘要 · Abstract (English)
Birds Eye View perception models require extensive data to perform and generalize effectively. While traditional datasets often provide abundant driving scenes from diverse locations, this is not always the case. It is crucial to maximize the utility of the available training data. With the advent of large foundation models such as DINOv2 and Metric3Dv2, a pertinent question arises: can these models be integrated into existing model architectures to not only reduce the required training data but surpass the performance of current models? We choose two model architectures in the vehicle segmentation domain to alter: Lift-Splat-Shoot, and Simple-BEV. For Lift-Splat-Shoot, we explore the implementation of frozen DINOv2 for feature extraction and Metric3Dv2 for depth estimation, where we greatly exceed the baseline results by 7.4 IoU while utilizing only half the training data and iterations. Furthermore, we introduce an innovative application of Metric3Dv2's depth information as a PseudoLiDAR point cloud incorporated into the Simple-BEV architecture, replacing traditional LiDAR. This integration results in a +3 IoU improvement compared to the Camera-only model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。