用扩散模型实现真实人像的逼真光照重渲染
SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces
- 将光照重渲染视为基于物理引擎的图像重渲染问题
- 在真实照片上生成带高光和投影的自然光照效果
- 适合需要高质量人像光照编辑的视觉创作场景
本文提出SynthLight，一种用于人像光照重渲染的扩散模型。该方法将图像重渲染视为光照条件变化下的像素变换问题，利用基于物理的渲染引擎，通过3D头部资产在不同光照条件下合成数据集以模拟这一变换过程。为弥合合成与真实图像域之间的差距，提出两种策略：(1) 多任务训练，利用无光照标签的真实人像；(2) 基于无分类器引导的推理时扩散采样，借助输入人像更好保留细节。该方法可泛化至多样化真实照片，在保留身份特征的同时生成包含镜面高光与投射阴影的逼真光照效果。在Light Stage数据上的定量实验表明性能达到当前最优水平；在真实场景图像上的定性结果展现出前所未有的丰富光照效果。
原文摘要 · Abstract (English)
We introduce SynthLight, a diffusion model for portrait relighting. Our approach frames image relighting as a re-rendering problem, where pixels are transformed in response to changes in environmental lighting conditions. Using a physically-based rendering engine, we synthesize a dataset to simulate this lighting-conditioned transformation with 3D head assets under varying lighting. We propose two training and inference strategies to bridge the gap between the synthetic and real image domains: (1) multi-task training that takes advantage of real human portraits without lighting labels; (2) an inference time diffusion sampling procedure based on classifier-free guidance that leverages the input portrait to better preserve details. Our method generalizes to diverse real photographs and produces realistic illumination effects, including specular highlights and cast shadows, while preserving the subject's identity. Our quantitative experiments on Light Stage data demonstrate results comparable to state-of-the-art relighting methods. Our qualitative results on in-the-wild images showcase rich and unprecedented illumination effects. Project Page: \url{https://vrroom.github.io/synthlight/}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。