首个面向城市场景大视角外推的渲染基准,揭示现有方法泛化能力不足。
Extrapolated Urban View Synthesis Benchmark
- 构建首个基于多车多视角数据的城市外推视图合成基准
- 实测显示当前方法在大幅视角变化下性能显著下降,存在过拟合问题
- 适合自驾车与城市机器人仿真研究者参考,推动更鲁棒的视觉生成技术
逼真的模拟器对视觉主导型自动驾驶车辆(AV)的训练与评估至关重要。其核心是新视角合成(NVS),可生成多样未见视角以应对自动驾驶车辆连续且广泛的姿态分布。近年来,基于辐射场的方法(如3D高斯泼溅)实现了实时逼真渲染,被广泛用于大规模驾驶场景建模。然而,其性能通常在高度相关训练与测试视角的插值设置下评估。相比之下,测试视角与训练视角显著偏离的外推场景仍缺乏探索,限制了通用仿真技术的发展。为此,我们利用公开的带有多次遍历、多车辆和多摄像头的自动驾驶数据集,构建首个外推城市视图合成(EUVS)基准。同时,我们在不同评估设置下对主流NVS方法进行定量与定性评估。结果表明,当前NVS方法易过拟合训练视角;引入扩散先验或改进几何结构也无法根本提升大视角变化下的表现,凸显需要更鲁棒的方法与大规模训练。我们将发布数据集,助力自动驾驶与城市机器人仿真技术发展。
原文摘要 · Abstract (English)
Photorealistic simulators are essential for the training and evaluation of vision-centric autonomous vehicles (AVs). At their core is Novel View Synthesis (NVS), a crucial capability that generates diverse unseen viewpoints to accommodate the broad and continuous pose distribution of AVs. Recent advances in radiance fields, such as 3D Gaussian Splatting, achieve photorealistic rendering at real-time speeds and have been widely used in modeling large-scale driving scenes. However, their performance is commonly evaluated using an interpolated setup with highly correlated training and test views. In contrast, extrapolation, where test views largely deviate from training views, remains underexplored, limiting progress in generalizable simulation technology. To address this gap, we leverage publicly available AV datasets with multiple traversals, multiple vehicles, and multiple cameras to build the first Extrapolated Urban View Synthesis (EUVS) benchmark. Meanwhile, we conduct both quantitative and qualitative evaluations of state-of-the-art NVS methods across different evaluation settings. Our results show that current NVS methods are prone to overfitting to training views. Besides, incorporating diffusion priors and improving geometry cannot fundamentally improve NVS under large view changes, highlighting the need for more robust approaches and large-scale training. We will release the data to help advance self-driving and urban robotics simulation technology.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。