用无3D数据的渐进式蒸馏,让文生3D更快更准。
Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Data
- 通过多视角扩散模型蒸馏纹理与几何,无需真实3D标签训练
- 生成1.2秒完成高质量3D网格,比现有方法快且更稳定
- 仅增加2.5%参数量,适合快速部署在文本生成3D场景
从文本提示中秒级生成高质量3D网格极具吸引力。尽管近期工作尝试将预训练文生图扩散模型(如Stable Diffusion, SD)适配为3D生成器(如Triplane),但因缺乏高质量3D训练数据而常导致生成质量不佳。为此,我们提出一种新训练方案——渐进式渲染蒸馏(Progressive Rendering Distillation, PRD),通过蒸馏多视角扩散模型(如MVDream、RichDreamer)中的文本一致纹理与几何信息,使SD直接成为原生3D生成器,完全无需3D真值数据。训练中,PRD利用U-Net逐步去噪潜在表示,并在每步解码为3D输出。由于无需3D真值,可轻松扩大训练规模,提升对创意性复杂文本的生成质量。同时,该方法可将生成速度加速至仅数步。基于此,我们训练出名为TriplaneTurbo的Triplane生成器,仅增加2.5%可训练参数,可在1.2秒内生成高质量3D网格,显著优于此前文本到3D生成器,在效率与泛化能力上均占优。代码已开源:https://github.com/theEricMa/TriplaneTurbo。
原文摘要 · Abstract (English)
It is highly desirable to obtain a model that can generate high-quality 3D meshes from text prompts in just seconds. While recent attempts have adapted pre-trained text-to-image diffusion models, such as Stable Diffusion (SD), into generators of 3D representations (e.g., Triplane), they often suffer from poor quality due to the lack of sufficient high-quality 3D training data. Aiming at overcoming the data shortage, we propose a novel training scheme, termed as Progressive Rendering Distillation (PRD), eliminating the need for 3D ground-truths by distilling multi-view diffusion models and adapting SD into a native 3D generator. In each iteration of training, PRD uses the U-Net to progressively denoise the latent from random noise for a few steps, and in each step it decodes the denoised latent into 3D output. Multi-view diffusion models, including MVDream and RichDreamer, are used in joint with SD to distill text-consistent textures and geometries into the 3D outputs through score distillation. Since PRD supports training without 3D ground-truths, we can easily scale up the training data and improve generation quality for challenging text prompts with creative concepts. Meanwhile, PRD can accelerate the inference speed of the generation model in just a few steps. With PRD, we train a Triplane generator, namely TriplaneTurbo, which adds only $2.5\%$ trainable parameters to adapt SD for Triplane generation. TriplaneTurbo outperforms previous text-to-3D generators in both efficiency and quality. Specifically, it can produce high-quality 3D meshes in 1.2 seconds and generalize well for challenging text input. The code is available at https://github.com/theEricMa/TriplaneTurbo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。