arXiv:2503.18438cs.CV2025-03ICCV被引 39

提升自动驾驶仿真中生成图像的真实感,尤其改善地面等结构的渲染质量。

ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representation

  • 引入可学习空间形变网络,缩小生成视图与真实传感器数据的差异。
  • 在3D高斯中保留几何先验,优化外观属性同时保持结构完整。
  • 在多个数据集上显著提升地面重建精度,适合高保真仿真研究者。

将重建模型与生成模型结合已成为自动驾驶闭环仿真中的有前景范式。例如,ReconDreamer已在大规模驾驶动作渲染中表现卓越。然而,生成数据与真实传感器观测之间仍存在显著差距,特别是在结构化元素(如地面表面)的保真度方面。为此,我们提出ReconDreamer++,通过缓解域差距并优化地面表面表示,显著提升整体渲染质量。具体而言,ReconDreamer++引入新型轨迹可变形网络(NTDNet),利用可学习的空间形变机制弥合合成新视角与原始观测之间的域差距。针对地面等结构化元素,我们在3D高斯中保留几何先验知识,优化过程聚焦于细化外观属性,同时保持底层几何结构。在Waymo、nuScenes、PandaSet和EUVS等多个数据集上的实验验证了其优越性能。在Waymo数据集上,ReconDreamer++在原轨迹上表现接近Street Gaussians,而在新轨迹上显著优于ReconDreamer,具体提升包括NTA-IoU提高6.1%、FID改善23.0%、地面指标NTL-IoU提升4.5%,充分证明其在准确重建道路等结构化元素方面的有效性。

原文摘要 · Abstract (English)

Combining reconstruction models with generative models has emerged as a promising paradigm for closed-loop simulation in autonomous driving. For example, ReconDreamer has demonstrated remarkable success in rendering large-scale maneuvers. However, a significant gap remains between the generated data and real-world sensor observations, particularly in terms of fidelity for structured elements, such as the ground surface. To address these challenges, we propose ReconDreamer++, an enhanced framework that significantly improves the overall rendering quality by mitigating the domain gap and refining the representation of the ground surface. Specifically, ReconDreamer++ introduces the Novel Trajectory Deformable Network (NTDNet), which leverages learnable spatial deformation mechanisms to bridge the domain gap between synthesized novel views and original sensor observations. Moreover, for structured elements such as the ground surface, we preserve geometric prior knowledge in 3D Gaussians, and the optimization process focuses on refining appearance attributes while preserving the underlying geometric structure. Experimental evaluations conducted on multiple datasets (Waymo, nuScenes, PandaSet, and EUVS) confirm the superior performance of ReconDreamer++. Specifically, on Waymo, ReconDreamer++ achieves performance comparable to Street Gaussians for the original trajectory while significantly outperforming ReconDreamer on novel trajectories. In particular, it achieves substantial improvements, including a 6.1% increase in NTA-IoU, a 23. 0% improvement in FID, and a remarkable 4.5% gain in the ground surface metric NTL-IoU, highlighting its effectiveness in accurately reconstructing structured elements such as the road surface.

自动驾驶图像生成3D重建仿真

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。