通过跨视图先验增强扩散过程,生成高保真细节丰富的3D人像。
Towards High-Fidelity 3D Portrait Generation with Rich Details by Cross-View Prior-Aware Diffusion
- 在条件输入和扩散过程中同时引入多视角先验,提升视图一致性。
- 相比现有方法,生成的3D人像纹理更清晰,几何结构更准确。
- 适合需要高质量单图生成3D人像的场景,如数字人、虚拟偶像。
基于扩散模型的单图3D人像生成方法通常采用2D扩散模型提供多视角知识,并将其提炼为3D表示。然而,这些方法常难以生成高保真3D模型,导致纹理过度模糊。我们归因于扩散过程中对跨视图一致性的考虑不足,造成不同视角间差异显著,最终导致3D表示模糊。本文通过在条件输入与扩散过程双重利用多视角先验,生成一致且细节丰富的3D人像。从条件设计出发,提出混合先验扩散模型(Hybrid Priors Diffusion),显式与隐式结合多视角先验作为条件,增强多视角人像的一致性。从扩散角度,鉴于扩散噪声分布对细节纹理生成的重要影响,提出多视角噪声重采样策略,在优化过程中融合跨视图先验以增强表示一致性。大量实验表明,本方法可从单张图像生成几何准确、细节丰富的3D人像。项目页面见:https://haoran-wei.github.io/Portrait-Diffusion。
原文摘要 · Abstract (English)
Recent diffusion-based Single-image 3D portrait generation methods typically employ 2D diffusion models to provide multi-view knowledge, which is then distilled into 3D representations. However, these methods usually struggle to produce high-fidelity 3D models, frequently yielding excessively blurred textures. We attribute this issue to the insufficient consideration of cross-view consistency during the diffusion process, resulting in significant disparities between different views and ultimately leading to blurred 3D representations. In this paper, we address this issue by comprehensively exploiting multi-view priors in both the conditioning and diffusion procedures to produce consistent, detail-rich portraits. From the conditioning standpoint, we propose a Hybrid Priors Diffsion model, which explicitly and implicitly incorporates multi-view priors as conditions to enhance the status consistency of the generated multi-view portraits. From the diffusion perspective, considering the significant impact of the diffusion noise distribution on detailed texture generation, we propose a Multi-View Noise Resamplig Strategy integrated within the optimization process leveraging cross-view priors to enhance representation consistency. Extensive experiments demonstrate that our method can produce 3D portraits with accurate geometry and rich details from a single image. The project page is at \url{https://haoran-wei.github.io/Portrait-Diffusion}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。