用2D编辑思路生成3D内容,提升真实感与细节丰富度
Text-to-3D Generation by 2D Editing
- 基于2D编辑框架,融合噪声轨迹与文本引导去噪,分步提取多粒度信息
- 生成的3D模型在视觉质量上显著优于传统方法,保留丰富纹理与结构细节
- 适合需要高保真3D内容的创作者,如游戏、影视及设计领域从业者
从预训练的2D扩散模型中提炼3D表示,对游戏、电影和室内设计等领域的3D创作至关重要。当前基于SDS的方法受限于扩散模型中信息提炼效率低,难以生成逼真的3D内容。本文重新审视SDS方法的本质,发现其作为基础图像编辑过程,常导致过饱和、过度平滑、内容贫乏和多样性不足,根源在于单步去噪质量差。为此,我们提出一种新方法——3D生成通过编辑(GE3D)。GE3D每轮迭代采用2D编辑框架,结合保留输入图像信息的噪声轨迹与文本引导的去噪轨迹,并对齐双轨迹中的潜在表示。该方法通过多步去噪充分挖掘预训练扩散模型中的多粒度信息,实现高质量、逼真的3D输出。理论与实验结果均验证了方法的有效性,不仅推动3D生成技术发展,还建立了3D生成与2D编辑的新关联,或可激发后续研究。代码与演示已公开于https://jahnsonblack.github.io/GE3D/
原文摘要 · Abstract (English)
Distilling 3D representations from pretrained 2D diffusion models is essential for 3D creative applications across gaming, film, and interior design. Current SDS-based methods are hindered by inefficient information distillation from diffusion models, which prevents the creation of photorealistic 3D contents. In this paper, we first reevaluate the SDS approach by analyzing its fundamental nature as a basic image editing process that commonly results in over-saturation, over-smoothing, lack of rich content and diversity due to the poor-quality single-step denoising. In light of this, we then propose a novel method called 3D Generation by Editing (GE3D). Each iteration of GE3D utilizes a 2D editing framework that combines a noising trajectory to preserve the information of the input image, alongside a text-guided denoising trajectory. We optimize the process by aligning the latents across both trajectories. This approach fully exploits pretrained diffusion models to distill multi-granularity information through multiple denoising steps, resulting in photorealistic 3D outputs. Both theoretical and experimental results confirm the effectiveness of our approach, which not only advances 3D generation technology but also establishes a novel connection between 3D generation and 2D editing. This could potentially inspire further research in the field. Code and demos are released at https://jahnsonblack.github.io/GE3D/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。