用结构引导扩散模型，精准去除人像阴影并保留细节。
Structure-Guided Diffusion Models for High-Fidelity Portrait Shadow Removal
- 先提取无阴影结构图，再用其指导扩散模型去阴影。
- 在多个基准数据集上优于现有方法，无身份篡改、残影等问题。
- 特别适合需要高保真人像修复的图像编辑场景。
我们提出一种基于扩散模型的人像阴影去除方法，可鲁棒地生成高保真结果。不同于以往方法，我们将阴影去除建模为基于扩散的图像修复任务。首先，在包含多种合成光照条件的真实人像数据集上训练一个与阴影无关的结构提取网络，生成包含面部细节但排除阴影边界的结构图。该结构图作为条件，用于训练结构引导的图像修复扩散模型，以生成式方式去除阴影。最后，为恢复结构图未捕捉的细粒度细节（如睫毛、痣和斑点），我们利用阴影区域内的梯度作为引导，训练一个细节修复扩散模型，对去阴影结果进行精细化优化。大量实验表明，该方法显著优于现有方法，有效避免了人脸身份篡改、阴影残留、色彩失真、结构模糊及细节丢失等常见问题。代码已公开于 https://github.com/wanchang-yu/Structure-Guided-Diffusion-for-Portrait-Shadow-Removal。
原文摘要 · Abstract (English)
We present a diffusion-based portrait shadow removal approach that can robustly produce high-fidelity results. Unlike previous methods, we cast shadow removal as diffusion-based inpainting. To this end, we first train a shadow-independent structure extraction network on a real-world portrait dataset with various synthetic lighting conditions, which allows to generate a shadow-independent structure map including facial details while excluding the unwanted shadow boundaries. The structure map is then used as condition to train a structure-guided inpainting diffusion model for removing shadows in a generative manner. Finally, to restore the fine-scale details (e.g., eyelashes, moles and spots) that may not be captured by the structure map, we take the gradients inside the shadow regions as guidance and train a detail restoration diffusion model to refine the shadow removal result. Extensive experiments on the benchmark datasets show that our method clearly outperforms existing methods, and is effective to avoid previously common issues such as facial identity tampering, shadow residual, color distortion, structure blurring, and loss of details. Our code is available at https://github.com/wanchang-yu/Structure-Guided-Diffusion-for-Portrait-Shadow-Removal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。