arXiv:2409.00991cs.CVcs.AI2024-09中稿 · ACM MM 2024, and t…被引 13

用3D人脸先验引导扩散模型,提升模糊人脸修复的清晰度与身份保真度。

3D Priors-Guided Diffusion for Blind Face Restoration

  • 将3D人脸模型作为结构和身份约束嵌入去噪扩散过程
  • 在真实与合成数据集上均优于当前最优方法
  • 适合关注人脸修复质量与身份一致性的研究者

盲人脸修复旨在从退化图像中恢复清晰人脸。近期基于生成对抗网络(GAN)作为先验的方法取得了显著进展,但在复杂退化场景下难以平衡真实感与保真度。为此,我们提出一种新型扩散框架,将3D人脸先验作为结构和身份约束嵌入去噪扩散过程。具体而言,利用预训练恢复网络初步恢复的人脸图像,通过3D可变形模型(3DMM)重建3D人脸;采用定制的多层级特征提取方法,挖掘3D人脸的结构与身份信息,并将其映射至噪声估计过程。为进一步增强身份信息融合,提出时间感知融合模块(TAFB),该模块考虑扩散过程中从结构细化到纹理增强的动态特性,实现更高效自适应的权重融合。大量实验表明,本方法在合成与真实世界数据集上的盲人脸修复任务中均优于现有先进算法。代码已开源。

原文摘要 · Abstract (English)

Blind face restoration endeavors to restore a clear face image from a degraded counterpart. Recent approaches employing Generative Adversarial Networks (GANs) as priors have demonstrated remarkable success in this field. However, these methods encounter challenges in achieving a balance between realism and fidelity, particularly in complex degradation scenarios. To inherit the exceptional realism generative ability of the diffusion model and also constrained by the identity-aware fidelity, we propose a novel diffusion-based framework by embedding the 3D facial priors as structure and identity constraints into a denoising diffusion process. Specifically, in order to obtain more accurate 3D prior representations, the 3D facial image is reconstructed by a 3D Morphable Model (3DMM) using an initial restored face image that has been processed by a pretrained restoration network. A customized multi-level feature extraction method is employed to exploit both structural and identity information of 3D facial images, which are then mapped into the noise estimation process. In order to enhance the fusion of identity information into the noise estimation, we propose a Time-Aware Fusion Block (TAFB). This module offers a more efficient and adaptive fusion of weights for denoising, considering the dynamic nature of the denoising process in the diffusion model, which involves initial structure refinement followed by texture detail enhancement. Extensive experiments demonstrate that our network performs favorably against state-of-the-art algorithms on synthetic and real-world datasets for blind face restoration. The Code is released on our project page at https://github.com/838143396/3Diffusion.

人脸修复扩散模型3D先验

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。