用2D扩散模型生成3D物体，无需3D训练数据
DreamFusion: Text-to-3D using 2D Diffusion
- 用2D扩散模型作先验，优化3D神经辐射场
- 生成的3D模型可任意视角观看与光照调整
- 无需3D数据或修改原模型，适合快速原型设计
近期文本到图像合成的突破得益于在数十亿图像-文本对上训练的扩散模型。将这一方法拓展至3D合成需要大规模标注3D数据集和高效的3D去噪架构，但目前均不可得。本文通过使用预训练的2D文本到图像扩散模型实现文本到3D合成。我们提出基于概率密度蒸馏的损失函数，使2D扩散模型可作为参数化图像生成器优化的先验。在类似DeepDream的流程中，通过梯度下降优化一个随机初始化的3D模型（神经辐射场，NeRF），使其从任意角度的2D渲染图达到低损失。最终生成的3D模型可任意视角查看、任意光照重渲染或合成进任意3D环境。该方法无需3D训练数据，也无需修改图像扩散模型，证明了预训练图像扩散模型作为先验的有效性。
原文摘要 · Abstract (English)
Recent breakthroughs in text-to-image synthesis have been driven by diffusion models trained on billions of image-text pairs. Adapting this approach to 3D synthesis would require large-scale datasets of labeled 3D data and efficient architectures for denoising 3D data, neither of which currently exist. In this work, we circumvent these limitations by using a pretrained 2D text-to-image diffusion model to perform text-to-3D synthesis. We introduce a loss based on probability density distillation that enables the use of a 2D diffusion model as a prior for optimization of a parametric image generator. Using this loss in a DeepDream-like procedure, we optimize a randomly-initialized 3D model (a Neural Radiance Field, or NeRF) via gradient descent such that its 2D renderings from random angles achieve a low loss. The resulting 3D model of the given text can be viewed from any angle, relit by arbitrary illumination, or composited into any 3D environment. Our approach requires no 3D training data and no modifications to the image diffusion model, demonstrating the effectiveness of pretrained image diffusion models as priors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。