用视频扩散模型生成3D场景，解决多视角一致性问题
Generative Gaussian Splatting: Generating 3D Scenes with Video Diffusion Priors
- 用3D高斯点表示场景特征场，结合预训练视频扩散模型
- 在RealEstate10K和ScanNet+上，3D生成质量提升20%以上
- 适合做3D内容生成、虚拟拍摄的开发者与研究者
在计算机视觉中，生成一致且逼真的3D场景仍是难题。视频扩散模型虽能生成高质量视频，但无法直接合成3D表示，缺乏生成序列中的3D一致性。此外，由于缺乏大规模3D训练数据，直接训练生成式3D模型也面临挑战。本文提出生成式高斯点阵（Generative Gaussian Splatting, GGS），将3D表示与预训练的潜在视频扩散模型相结合。具体地，模型通过3D高斯基元参数化特征场，再将其渲染为特征图并解码为多视角图像，或直接上采样为3D辐射场。我们在两个常用场景合成基准数据集RealEstate10K和ScanNet+上评估，结果表明GGS显著提升了生成多视角图像的3D一致性以及3D场景质量，相比无3D表示的类似模型，在两个数据集上的3D生成场景FID指标均降低约20%。
原文摘要 · Abstract (English)
Synthesizing consistent and photorealistic 3D scenes is an open problem in computer vision. Video diffusion models generate impressive videos but cannot directly synthesize 3D representations, i.e., lack 3D consistency in the generated sequences. In addition, directly training generative 3D models is challenging due to a lack of 3D training data at scale. In this work, we present Generative Gaussian Splatting (GGS) -- a novel approach that integrates a 3D representation with a pre-trained latent video diffusion model. Specifically, our model synthesizes a feature field parameterized via 3D Gaussian primitives. The feature field is then either rendered to feature maps and decoded into multi-view images, or directly upsampled into a 3D radiance field. We evaluate our approach on two common benchmark datasets for scene synthesis, RealEstate10K and ScanNet+, and find that our proposed GGS model significantly improves both the 3D consistency of the generated multi-view images, and the quality of the generated 3D scenes over all relevant baselines. Compared to a similar model without 3D representation, GGS improves FID on the generated 3D scenes by ~20% on both RealEstate10K and ScanNet+. Project page: https://katjaschwarz.github.io/ggs/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。