通过选参考图、传特征、重建输出,提升人脸复原保真度。
RefSTAR: Blind Facial Image Restoration with Reference Selection, Transfer, and Reconstruction
- 构建参考图选择模块,基于1万对真实-参考图像训练。
- 设计特征融合机制,避免注意力机制导致的特征失真。
- 引入重建与循环一致性损失,确保参考特征有效保留。
由于退化复杂且人脸敏感,盲人脸图像复原极具挑战。现有方法虽借助生成先验或高质量参考图,仍面临身份保持难题,主要因细节纹理特征引入不当。本文提出新方法RefSTAR,聚焦参考图的选择、特征迁移与重建。在选择阶段,构建包含10,000对真实-参考图像标注掩码的RefSel-HQ数据集;在迁移阶段,设计特征融合范式,避免原始交叉注意力的平凡解;最后提出参考图重建机制,并结合掩码重新设计循环一致性损失,强化参考特征在输出中的存在。在多种骨干模型上实验均显示更优性能,显著提升身份保真度与参考特征传递质量。代码、数据集及预训练模型已开源。
原文摘要 · Abstract (English)
Blind facial image restoration is highly challenging due to unknown complex degradations and the sensitivity of humans to faces. Although existing methods introduce auxiliary information from generative priors or high-quality reference images, they still struggle with identity preservation problems, mainly due to improper feature introduction on detailed textures. In this paper, we focus on effectively incorporating appropriate features from high-quality reference images, presenting a novel blind facial image restoration method that considers reference selection, transfer, and reconstruction (RefSTAR). In terms of selection, we construct a reference selection (RefSel) module. For training the RefSel module, we construct a RefSel-HQ dataset through a mask generation pipeline, which contains annotating masks for 10,000 ground truth-reference pairs. As for the transfer, due to the trivial solution in vanilla cross-attention operations, a feature fusion paradigm is designed to force the features from the reference to be integrated. Finally, we propose a reference image reconstruction mechanism that further ensures the presence of reference image features in the output image. The cycle consistency loss is also redesigned in conjunction with the mask. Extensive experiments on various backbone models demonstrate superior performance, showing better identity preservation ability and reference feature transfer quality. Source code, dataset, and pre-trained models are available at https://github.com/yinzhicun/RefSTAR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。