用语义引导的两阶段生成模型,修复大范围人脸缺失区域,保持身份和结构一致。
Semantic-Guided Two-Stage GAN for Face Inpainting with Hybrid Perceptual Encoding
- 分两阶段:先建语义布局,再细化纹理
- 在CelebA-HQ和FFHQ上提升LPIPS、PSNR、SSIM指标
- 无需针对特定遮罩训练,可处理任意形状遮挡
人脸图像修复旨在恢复人脸图像中缺失或损坏区域,同时保持身份一致性、结构连贯性和逼真图像质量,专门用于照片修复。尽管深度生成模型近年来取得诸多进展,现有方法在处理大范围不规则遮罩时仍存在边缘模糊、语义不一致或面部结构不合理等问题,主要源于直接像素级合成及对人脸先验信息利用不足。本文提出一种新型架构,通过语义引导的分层合成解决上述挑战。第一阶段融合卷积神经网络(CNN)的局部特征与视觉变换器(Vision Transformer)的全局特征,生成清晰详细的语义布局;第二阶段采用多模态纹理生成器,从多尺度提取信息以精修布局,确保整体一致。该架构通过动态注意力机制自然适应任意遮罩配置,无需特定遮罩训练。在CelebA-HQ和FFHQ两个数据集上的实验表明,模型优于当前最先进方法,在LPIPS、PSNR和SSIM等指标上均有提升,尤其在大范围修复场景下表现更优,视觉效果显著。
原文摘要 · Abstract (English)
Facial Image inpainting aim is to restore the missing or corrupted regions in face images while preserving identity, structural consistency and photorealistic image quality, a task specifically created for photo restoration. Though there are recent lot of advances in deep generative models, existing methods face problems with large irregular masks, often producing blurry textures on the edges of the masked region, semantic inconsistencies, or unconvincing facial structures due to direct pixel level synthesis approach and limited exploitation of facial priors. In this paper we propose a novel architecture, which address these above challenges through semantic-guided hierarchical synthesis. Our approach starts with a method that organizes and synthesizes information based on meaning, followed by refining the texture. This process gives clear insights into the facial structure before we move on to creating detailed images. In the first stage, we blend two techniques: one that focuses on local features with CNNs and global features with Vision Transformers. This helped us create clear and detailed semantic layouts. In the second stage, we use a Multi-Modal Texture Generator to refine these layouts by pulling in information from different scales, ensuring everything looks cohesive and consistent. The architecture naturally handles arbitrary mask configurations through dynamic attention without maskspecific training. Experiment on two datasets CelebA-HQ and FFHQ shows that our model outperforms other state-of-the-art methods, showing improvements in metrics like LPIPS, PSNR, and SSIM. It produces visually striking results with better semantic preservation, in challenging large-area inpainting situations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。