arXiv:2505.23462cs.CV2025-05中稿 · ed

用轻量适配器对齐低质人脸的潜在表示,提升扩散模型修复效果。

LAFR: Efficient Diffusion-based Blind Face Restoration via Latent Codebook Alignment Adapter

  • 设计代码本适配器,将低质图像潜在向量对齐到扩散先验分布内。
  • 仅用0.9%的FFHQ数据微调,训练时间减少70%,性能媲美昂贵方法。
  • 融合语义身份与面部结构约束,修复后人脸更保真且结构一致。

从低质量图像中进行盲人脸修复是一项挑战性任务,既要保证高保真重建,又要保留人脸身份特征。尽管像Stable Diffusion这样的扩散模型在生成高质量图像方面表现优异,但其VAE模块通常在以高质量内容为主的自然图像数据上训练;严重退化的低质量输入会导致潜在向量落在扩散先验的低密度区域,削弱去噪过程中的条件控制能力。现有方法多通过重训练VAE编码器来解决此问题,但计算成本高、内存消耗大。为此,我们提出LAFR,一种基于代码本的潜在空间适配器,可将低质量输入的潜在向量对齐至扩散先验所训练的分布区域内,实现无需修改原始VAE的更好条件采样。为进一步提升相对于OSEDiff类基线的身份与结构一致性,引入多层级修复损失,结合语义身份嵌入与面部结构先验约束。此外,利用人脸图像固有的结构规律,仅对0.9%的FFHQ数据进行轻量微调,即可达到竞争性效果,同时将训练时间减少70%。我们还简要分析了为何代码本修正能弥合模式依赖的潜在间隙,而全局归一化或仿射变换无法做到。综上,LAFR是一种数据与计算高效的单步人脸修复方法,在感知质量、FID、关键点准确率及身份/结构一致性方面优于OSEDiff类基线,且不追求统一最先进水平,但在保持竞争力的同时显著优于更昂贵的流程。

原文摘要 · Abstract (English)

Blind face restoration from low-quality images is a challenging task that requires not only high-fidelity image reconstruction, but also preservation of facial identity. Although diffusion models like Stable Diffusion have shown promise in generating high-quality images, their VAE modules are typically trained on broad natural-image data dominated by HQ content; severely degraded LQ inputs therefore yield latents that fall in low-density regions of the diffusion prior, weakening the effectiveness of LQ conditions during the denoising process. Existing approaches often tackle this issue by retraining the VAE encoder, which is computationally expensive and memory intensive. To address this limitation efficiently, we propose LAFR, a codebook-based latent space adapter that aligns LQ latents to the in-distribution region the diffusion prior was trained to denoise, enabling sampling from a better matched latent condition without altering the original VAE. To further improve identity and structural consistency relative to OSEDiff-style baselines, we introduce a multilevel restoration loss combining constraints from semantic identity embeddings and facial structural priors. Furthermore, by leveraging the inherent structural regularity of facial images, we show that lightweight fine-tuning of a diffusion prior on just 0.9% of FFHQ can achieve competitive results while reducing training time by 70%. We also give a short analysis of why a codebook correction closes a mode-dependent latent gap that no global normalisation or affine map can. In summary, LAFR is a data and compute efficient one-step face-restoration method that improves perceptual quality, FID, landmark accuracy, and identity/structure consistency over OSEDiff-style baselines, without claiming uniform state of the art; it reaches this operating point while remaining competitive with far more expensive pipelines.

人脸修复扩散模型潜在对齐轻量微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。