用4D高斯生成驾驶场景，实现高质量多视角视频合成
WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Driving
- 基于多模态潜空间扩散模型，前馈生成像素对齐的4D高斯点
- 在多个基准数据集上实现高保真、时空一致的多轨道新视角视频
- 适合需要可控驾驶数据生成的研究者和自动驾驶系统开发者
近期驾驶场景生成与重建技术在提升自动驾驶系统训练数据的可扩展性和可控性方面展现出巨大潜力。现有生成方法主要聚焦于合成多样且高保真的驾驶视频，但受限于3D一致性不足和视角覆盖稀疏，难以支持高质量的新视角合成（NVS）。相反，现有的3D/4D重建方法虽显著提升了真实驾驶场景的新视角合成质量，却缺乏生成能力。为解决生成与重建之间的矛盾，我们提出WorldSplat，一种面向4D驾驶场景生成的前馈式框架。该方法通过两个关键步骤实现：(i) 提出一种4D感知的潜在扩散模型，融合多模态信息，以前馈方式生成像素对齐的4D高斯；(ii) 进而利用增强的视频扩散模型优化由这些高斯渲染出的新视角视频。在多个基准数据集上的大量实验表明，WorldSplat能有效生成高保真、时空一致的多轨道新视角驾驶视频。
原文摘要 · Abstract (English)
Recent advances in driving-scene generation and reconstruction have demonstrated significant potential for enhancing autonomous driving systems by producing scalable and controllable training data. Existing generation methods primarily focus on synthesizing diverse and high-fidelity driving videos; however, due to limited 3D consistency and sparse viewpoint coverage, they struggle to support convenient and high-quality novel-view synthesis (NVS). Conversely, recent 3D/4D reconstruction approaches have significantly improved NVS for real-world driving scenes, yet inherently lack generative capabilities. To overcome this dilemma between scene generation and reconstruction, we propose WorldSplat, a novel feed-forward framework for 4D driving-scene generation. Our approach effectively generates consistent multi-track videos through two key steps: (i) We introduce a 4D-aware latent diffusion model integrating multi-modal information to produce pixel-aligned 4D Gaussians in a feed-forward manner. (ii) Subsequently, we refine the novel view videos rendered from these Gaussians using a enhanced video diffusion model. Extensive experiments conducted on benchmark datasets demonstrate that WorldSplat effectively generates high-fidelity, temporally and spatially consistent multi-track novel view driving videos. Project: https://wm-research.github.io/worldsplat/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。