用3D生成粗略场景，再用2D细化细节，实现可控的城市场景生成。
ScenDi: 3D-to-2D Scene Diffusion Cascades for Urban Generation
- 先用3D扩散模型生成3D高斯点云，支持框体、地图或文本条件输入
- 再用2D视频扩散模型基于3D渲染图增强细节，保持相机轨迹准确
- 在Waymo和KITTI-360数据集上验证，兼具视觉质量与相机控制能力
近期基于扩散模型的3D物体生成取得显著进展，但真实城市场景的生成仍具挑战。仅依赖3D扩散模型的方法常导致外观细节退化，而仅使用2D扩散模型的方法则通常牺牲相机可控性。为此，我们提出ScenDi，一种融合3D与2D扩散模型的城市场景生成方法。首先训练一个3D潜在扩散模型生成3D高斯点云（3DGS），可低分辨率渲染图像；该过程可选地通过3D边界框、道路图或文本提示进行条件控制。随后，训练一个2D视频扩散模型，以3DGS渲染图像为条件，增强外观细节。通过利用粗略3D场景引导2D视频扩散，ScenDi能根据输入条件生成符合预期的场景，并精确遵循相机轨迹。在两个具有挑战性的真实世界数据集Waymo和KITTI-360上的实验表明该方法有效。
原文摘要 · Abstract (English)
Recent advancements in 3D object generation using diffusion models have achieved remarkable success, but generating realistic 3D urban scenes remains challenging. Existing methods relying solely on 3D diffusion models tend to suffer a degradation in appearance details, while those utilizing only 2D diffusion models typically compromise camera controllability. To overcome this limitation, we propose ScenDi, a method for urban scene generation that integrates both 3D and 2D diffusion models. We first train a 3D latent diffusion model to generate 3D Gaussians, enabling the rendering of images at a relatively low resolution. To enable controllable synthesis, this 3DGS generation process can be optionally conditioned by specifying inputs such as 3d bounding boxes, road maps, or text prompts. Then, we train a 2D video diffusion model to enhance appearance details conditioned on rendered images from the 3D Gaussians. By leveraging the coarse 3D scene as guidance for 2D video diffusion, ScenDi generates desired scenes based on input conditions and successfully adheres to accurate camera trajectories. Experiments on two challenging real-world datasets, Waymo and KITTI-360, demonstrate the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。