用扩散模型实现人脸表演的自由视角光照重渲染,细节逼真。
DifFRelight: Diffusion-Based Facial Performance Relighting
- 基于扩散模型,结合平面光与定向光信息进行光照控制。
- 可还原眼反射、次表面散射等复杂光影效果,保持皮肤纹理一致。
- 适合影视特效、虚拟人生成等需要高保真光照的应用场景。
我们提出一种基于扩散模型的自由视角人脸表演光照重渲染框架。利用包含多种表情和光照条件(包括平光及单光源)的特定个体数据集,训练扩散模型实现精确光照控制,从平光输入生成高质量重光照人脸图像。框架融合平光捕获的空间对齐信息与随机噪声,并引入集成光照信息实现全局调控,利用预训练的Stable Diffusion模型先验知识。该模型应用于统一平光环境下捕获的动态人脸表演,通过可扩展的动态3D高斯泼溅方法重建并合成新视角,保持重光照结果的质量与一致性。我们还引入新型面光源表示与方向光融合,实现光大小与方向的联合调节;并支持多方向光组合生成高动态范围成像(HDRI),在复杂光照下生成动态序列。评估表明,模型在多种表情间具有良好泛化能力,精准还原眼反射、次表面散射、自阴影与半透明等复杂光照效果,显著提升视觉真实性。
原文摘要 · Abstract (English)
We present a novel framework for free-viewpoint facial performance relighting using diffusion-based image-to-image translation. Leveraging a subject-specific dataset containing diverse facial expressions captured under various lighting conditions, including flat-lit and one-light-at-a-time (OLAT) scenarios, we train a diffusion model for precise lighting control, enabling high-fidelity relit facial images from flat-lit inputs. Our framework includes spatially-aligned conditioning of flat-lit captures and random noise, along with integrated lighting information for global control, utilizing prior knowledge from the pre-trained Stable Diffusion model. This model is then applied to dynamic facial performances captured in a consistent flat-lit environment and reconstructed for novel-view synthesis using a scalable dynamic 3D Gaussian Splatting method to maintain quality and consistency in the relit results. In addition, we introduce unified lighting control by integrating a novel area lighting representation with directional lighting, allowing for joint adjustments in light size and direction. We also enable high dynamic range imaging (HDRI) composition using multiple directional lights to produce dynamic sequences under complex lighting conditions. Our evaluations demonstrate the models efficiency in achieving precise lighting control and generalizing across various facial expressions while preserving detailed features such as skintexture andhair. The model accurately reproduces complex lighting effects like eye reflections, subsurface scattering, self-shadowing, and translucency, advancing photorealism within our framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。