用扩散模型直接生成3D高斯点云，单视角就能快速重建场景。
Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation and Reconstruction
- 在每一步扩散过程中直接输出3D高斯点云，保证多视角一致性。
- 物体和场景的PSNR/FID分别提升2.20/23.25和1.34/19.16，无需深度估计。
- 速度超5倍快（A100上约6秒），适合交互式3D生成与重建。
现有前馈式图像转3D方法主要依赖2D多视角扩散模型，难以保证3D一致性，换视角时易崩溃，且多限于物体中心场景。本文提出新型单阶段3D扩散模型DiffusionGS，仅需单视图即可生成物体或重建场景。DiffusionGS在每一步扩散过程中直接输出3D高斯点云，强制保持视角一致性，使模型可稳定生成任意方向提示下的3D内容，突破物体中心限制。为提升模型能力与泛化性，我们设计了场景-物体混合训练策略以扩大3D训练数据规模。实验表明，相较于当前最优方法，DiffusionGS在物体与场景上的PSNR/FID分别提升2.20 dB/23.25和1.34 dB/19.16，且无需深度估计。同时，推理速度超过5倍加速（A100 GPU上约6秒）。项目主页展示视频与交互结果，代码与模型已公开。
原文摘要 · Abstract (English)
Existing feedforward image-to-3D methods mainly rely on 2D multi-view diffusion models that cannot guarantee 3D consistency. These methods easily collapse when changing the prompt view direction and mainly handle object-centric cases. In this paper, we propose a novel single-stage 3D diffusion model, DiffusionGS, for object generation and scene reconstruction from a single view. DiffusionGS directly outputs 3D Gaussian point clouds at each timestep to enforce view consistency and allow the model to generate robustly given prompt views of any directions, beyond object-centric inputs. Plus, to improve the capability and generality of DiffusionGS, we scale up 3D training data by developing a scene-object mixed training strategy. Experiments show that DiffusionGS yields improvements of 2.20 dB/23.25 and 1.34 dB/19.16 in PSNR/FID for objects and scenes than the state-of-the-art methods, without depth estimator. Plus, our method enjoys over 5$\times$ faster speed ($\sim$6s on an A100 GPU). Our Project page at https://caiyuanhao1998.github.io/project/DiffusionGS/ shows the video and interactive results. The code and models are publicly available at https://github.com/caiyuanhao1998/Open-DiffusionGS
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。