用生成先验实现单图人脸光照重演,效果更真实且保身份。
3DPR: Single Image 3D Portrait Relight using Generative Priors
- 基于生成模型潜空间与光场数据,学习高频率面部反照率先验
- 仅需少量光场数据即可训练,生成的OLAT图像支持物理级环境光照重演
- 适合需要高质量人脸光照编辑的应用,如影视特效、虚拟形象
给定单张人像图片,生成全新光照下的三维人脸视图是一个本质欠约束问题。传统图形学方法通过可微渲染显式分解几何、材质和光照,但受限于模型假设与参数化近似。本文提出3DPR,一种基于生成先验的图像重光照模型,利用在光场装置中采集的多视角单光源(OLAT)图像学习高质量高频率面部反照率分布。我们构建了包含139名受试者的大型4K多视角OLAT数据集。利用预训练生成头像模型的潜空间,提供来自真实场景图像的数据先验。输入图像通过编码器逆映射嵌入该潜空间。随后,基于新提出的三平面反射网络,在光场数据上训练以合成高保真OLAT图像,实现图像级重光照。该网络在生成头像模型的潜空间中运行,使少量光场图像即可有效训练。结合给定的HDRI环境图合成多角度光照结果,实现物理准确的环境重光照。定量与定性评估表明,3DPR优于先前方法,尤其在保持身份一致性和捕捉镜面高光、自阴影、次表面散射等光照细节方面表现优异。
原文摘要 · Abstract (English)
Rendering novel, relit views of a human head, given a monocular portrait image as input, is an inherently underconstrained problem. The traditional graphics solution is to explicitly decompose the input image into geometry, material and lighting via differentiable rendering; but this is constrained by the multiple assumptions and approximations of the underlying models and parameterizations of these scene components. We propose 3DPR, an image-based relighting model that leverages generative priors learnt from multi-view One-Light-at-A-Time (OLAT) images captured in a light stage. We introduce a new diverse and large-scale multi-view 4K OLAT dataset of 139 subjects to learn a high-quality prior over the distribution of high-frequency face reflectance. We leverage the latent space of a pre-trained generative head model that provides a rich prior over face geometry learnt from in-the-wild image datasets. The input portrait is first embedded in the latent manifold of such a model through an encoder-based inversion process. Then a novel triplane-based reflectance network trained on our lightstage data is used to synthesize high-fidelity OLAT images to enable image-based relighting. Our reflectance network operates in the latent space of the generative head model, crucially enabling a relatively small number of lightstage images to train the reflectance model. Combining the generated OLATs according to a given HDRI environment maps yields physically accurate environmental relighting results. Through quantitative and qualitative evaluations, we demonstrate that 3DPR outperforms previous methods, particularly in preserving identity and in capturing lighting effects such as specularities, self-shadows, and subsurface scattering. Project Page: https://vcai.mpi-inf.mpg.de/projects/3dpr/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。