arXiv:2412.01717cs.CV2024-12ICCV被引 8

用生成先验提升任意路径驾驶视角合成质量,实现实时高保真模拟。

Driving View Synthesis on Free-form Trajectories with Generative Prior

  • 通过扩散模型修复3D高斯的渲染缺陷,反向优化高斯模型。
  • 在新路径上实现高质量实时视角合成,无需预设退化模式。
  • 适合自动驾驶仿真、自由轨迹可视化等场景,尤其擅长复杂路径生成。

沿自由轨迹的驾驶视角合成对真实驾驶模拟至关重要,支持端到端驾驶策略的闭环评估。现有方法在记录路径上表现良好,但在新轨迹上泛化能力弱,因驾驶视频视角有限。为此,我们提出DriveX框架,将生成先验逐步融入3D高斯模型优化中。利用视频扩散模型修复训练中高斯模型生成的新轨迹渲染结果,修复后的视频再作为额外监督信号反向优化3D高斯模型。具体地,设计基于图像修复的视频重建任务,分离退化区域识别与生成能力,避免在扩散模型训练中模拟特定退化模式。为进一步提升生成内容的一致性与保真度,伪真值随新轨迹渲染质量逐步更新,使两个组件协同优化并相互强化,同时最小化对优化过程的干扰。通过紧密融合3D场景表示与生成先验,DriveX实现了在未记录轨迹上的高质量实时视角合成,为自由轨迹的灵活、真实驾驶模拟开辟新可能。

原文摘要 · Abstract (English)

Driving view synthesis along free-form trajectories is essential for realistic driving simulations, enabling closed-loop evaluation of end-to-end driving policies. Existing methods excel at view interpolation along recorded paths but struggle to generalize to novel trajectories due to limited viewpoints in driving videos. To tackle this challenge, we propose DriveX, a novel free-form driving view synthesis framework, that progressively distills generative prior into the 3D Gaussian model during its optimization. Within this framework, we utilize a video diffusion model to refine the degraded novel trajectory renderings from the in-training Gaussian model, while the restored videos in turn serve as additional supervision for optimizing the 3D Gaussian. Concretely, we craft an inpainting-based video restoration task, which can disentangle the identification of degraded regions from the generative capability of the diffusion model and remove the need of simulating specific degraded pattern in the training of the diffusion model. To further enhance the consistency and fidelity of generated contents, the pseudo ground truth is progressively updated with gradually improved novel trajectory rendering, allowing both components to co-adapt and reinforce each other while minimizing the disruption on the optimization. By tightly integrating 3D scene representation with generative prior, DriveX achieves high-quality view synthesis beyond recorded trajectories in real time--unlocking new possibilities for flexible and realistic driving simulations on free-form trajectories.

视角合成生成模型自动驾驶3D高斯

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。