用统一的俯视图潜空间实现自动驾驶多视角图像一致生成
BEV-VAE: Multi-view Image Generation with Spatial Consistency for Autonomous Driving
- 构建多视角图像变分自编码器，学习统一的俯视图潜空间
- 在nuScenes和AV2数据集上实现高质量3D一致重建与生成
- 支持任意视角生成，可选输入3D布局控制生成过程
自动驾驶中的多视角图像生成需要跨摄像头视图保持一致的3D场景理解。现有方法多将此问题视为二维图像集生成任务，缺乏显式的3D建模。本文认为结构化表示对场景生成至关重要，尤其在自动驾驶应用中。提出BEV-VAE，实现一致且可控的视图合成。首先训练一个多视角图像变分自编码器，获得紧凑统一的俯视图（BEV）潜空间；随后使用潜空间扩散变换器生成场景。BEV-VAE支持根据相机配置生成任意视角图像，并可选输入3D布局。在nuScenes和Argoverse 2（AV2）数据集上的实验表明，该方法在3D一致性重建与生成方面表现优异。代码已开源：https://github.com/Czm369/bev-vae。
原文摘要 · Abstract (English)
Multi-view image generation in autonomous driving demands consistent 3D scene understanding across camera views. Most existing methods treat this problem as a 2D image set generation task, lacking explicit 3D modeling. However, we argue that a structured representation is crucial for scene generation, especially for autonomous driving applications. This paper proposes BEV-VAE for consistent and controllable view synthesis. BEV-VAE first trains a multi-view image variational autoencoder for a compact and unified BEV latent space and then generates the scene with a latent diffusion transformer. BEV-VAE supports arbitrary view generation given camera configurations, and optionally 3D layouts. Experiments on nuScenes and Argoverse 2 (AV2) show strong performance in both 3D consistent reconstruction and generation. The code is available at: https://github.com/Czm369/bev-vae.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。