用2D图像训练文本控制的3D生成模型，效果媲美高保真图像生成。
GANFusion: Feed-Forward Text-to-3D with Diffusion in GAN Space
- 先用GAN从单视角2D图生成3D特征，再用扩散模型学习文本条件采样。
- 仅需2D数据监督，生成的人体3D模型质量接近顶尖图像生成水平。
- 适合做高质量、可文本控制的3D角色生成，无需测试时优化。
我们训练了一个前馈式文本到3D的扩散生成器，仅使用单视角2D数据进行监督，用于生成人体角色。现有3D生成模型尚无法达到图像或视频生成模型的保真度。当前最先进的3D生成器通常依赖显式的3D监督，受限于现有3D数据的数量与多样性。而仅以2D数据为监督的生成器通常结果粗糙，无法实现文本条件控制，或需依赖测试时优化。我们观察到GAN与扩散模型具有互补优势：GAN可在仅2D监督下高效训练，生成高质量3D物体，但难以文本条件化；而去噪扩散模型能高效实现文本条件化，但难以仅用2D数据训练。为此，我们提出GANFusion：首先使用仅依赖单视角2D数据训练的GAN架构生成无条件的三平面（triplane）特征；然后对GAN生成的随机样本添加描述，训练一个文本条件扩散模型，直接学习从优质三平面特征空间中采样，这些特征可解码为3D对象。
原文摘要 · Abstract (English)
We train a feed-forward text-to-3D diffusion generator for human characters using only single-view 2D data for supervision. Existing 3D generative models cannot yet match the fidelity of image or video generative models. State-of-the-art 3D generators are either trained with explicit 3D supervision and are thus limited by the volume and diversity of existing 3D data. Meanwhile, generators that can be trained with only 2D data as supervision typically produce coarser results, cannot be text-conditioned, or must revert to test-time optimization. We observe that GAN- and diffusion-based generators have complementary qualities: GANs can be trained efficiently with 2D supervision to produce high-quality 3D objects but are hard to condition on text. In contrast, denoising diffusion models can be conditioned efficiently but tend to be hard to train with only 2D supervision. We introduce GANFusion, which starts by generating unconditional triplane features for 3D data using a GAN architecture trained with only single-view 2D data. We then generate random samples from the GAN, caption them, and train a text-conditioned diffusion model that directly learns to sample from the space of good triplane features that can be decoded into 3D objects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。