让自动驾驶模型无视摄像头位置变化,提升复杂场景下的行驶稳定性。
VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting
- 用3D高斯点云实现端到端驾驶,通过重建场景辅助视角合成。
- 在新视角下测试时,驾驶成功率提升18.7%,比基线更稳定。
- 适合做真实世界自动驾驶系统开发,尤其关注多车配置的场景。
端到端自动驾驶(E2E-AD)将感知、预测与规划整合为统一的数据驱动框架,但因车辆配置多样导致摄像头视角变化,其鲁棒性仍是难题。本文提出VR-Drive,一种新型E2E-AD框架,通过联合学习3D场景重建作为辅助任务,实现面向规划的视角合成。不同于以往依赖特定场景的合成方法,VR-Drive采用前馈推理策略,可在训练时从稀疏视角在线增强,无需额外标注。为提升视角一致性,引入视角混合记忆库以实现跨视角时间交互,并设计视角一致的知识蒸馏策略,将原始视角知识迁移至合成视角。整个模型全端到端训练,有效缓解合成噪声,改善视角切换下的规划性能。此外,我们发布一个新基准数据集,用于评估在新摄像头视角下的E2E-AD表现,支持全面分析。实验表明,VR-Drive是可扩展且鲁棒的解决方案,适用于真实世界的端到端自动驾驶部署。
原文摘要 · Abstract (English)
End-to-end autonomous driving (E2E-AD) has emerged as a promising paradigm that unifies perception, prediction, and planning into a holistic, data-driven framework. However, achieving robustness to varying camera viewpoints, a common real-world challenge due to diverse vehicle configurations, remains an open problem. In this work, we propose VR-Drive, a novel E2E-AD framework that addresses viewpoint generalization by jointly learning 3D scene reconstruction as an auxiliary task to enable planning-aware view synthesis. Unlike prior scene-specific synthesis approaches, VR-Drive adopts a feed-forward inference strategy that supports online training-time augmentation from sparse views without additional annotations. To further improve viewpoint consistency, we introduce a viewpoint-mixed memory bank that facilitates temporal interaction across multiple viewpoints and a viewpoint-consistent distillation strategy that transfers knowledge from original to synthesized views. Trained in a fully end-to-end manner, VR-Drive effectively mitigates synthesis-induced noise and improves planning under viewpoint shifts. In addition, we release a new benchmark dataset to evaluate E2E-AD performance under novel camera viewpoints, enabling comprehensive analysis. Our results demonstrate that VR-Drive is a scalable and robust solution for the real-world deployment of end-to-end autonomous driving systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。