用3D先验引导扩散模型,单图生成任意视角姿态的人像。
One-Shot Novel View and Pose Human Image Synthesis via 3D Prior Guided Diffusion Model

- 引入3D法向图和颜色提示作为几何与色彩条件,指导图像生成。
- 在多个数据集上超越现有方法,尤其对遮挡部位重建效果更优。
- 适合需要高保真人体图像生成的研究者或开发者使用。
本文解决单张图像生成任意视角与姿态的人体图像问题。现有方法依赖2D姿态关键点进行姿态迁移,或基于通用人体NeRF提取点级特征生成图像,但前者难以处理复杂姿态,后者在缺乏可靠特征时无法准确恢复被遮挡部位。为此,我们提出一种基于条件去噪扩散模型的新方法,将新视角与姿态合成分解为一系列条件去噪步骤。通过引入3D人体先验(3D法向图与颜色提示)作为几何与颜色条件,模型在多步扩散过程中实现高质量生成,包括遮挡与不可见区域。进一步设计自重构式定制优化模块,在新人物测试中提升细节质量。在多个公开数据集上的实验表明,本方法显著优于已有方法,并展现出更强的跨数据集泛化能力。代码将开源于https://github.com/Yankeegsj/3DPGDM。
原文摘要 · Abstract (English)
This paper addresses the challenge of one-shot novel view and pose human image synthesis. The existing methods transfer the reference human image to a target pose using a set of 2D pose keypoints or synthesize human images based on generalizable human NeRF which uses human model priors to extract point-wise features. However, pose transfer based methods can not handle complex human pose using ambiguous 2D pose as the condition, while generalizable human NeRFs may be inaccurate to recover occluded/invisiable human parts without extracted reliable features. To solve these problems, we propose a novel approach for novel view and pose synthesis from a singe human image via conditional denoising diffusion model. Our diffusion model divides the novel view and pose synthesis problem into a sequence of conditional denoising steps. Specifically, to generate humans with complex and arbitrary poses, we introduce 3D human priors, i.e., 3D normal map and color prompt, as geometry and color conditions into the generation process. By transferring the reference human into the target human with a series of diffusion steps, our diffusion model enables high-quality synthesis including the occluded/invisible parts. Further, we propose a self-reconstruction based customized refinement to enhance fine details when tested on novel persons.Experimental results on different public datasets demonstrate that our approach significantly outperforms previous methods and also shows better generalization ability across datasets. The code will be made publicly available at https://github.com/Yankeegsj/3DPGDM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。