用扩散模型统一实现高保真人脸替换,支持换发丝和配饰。
Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Models
- 将换脸重构为训练时自监督修复,提升身份保留与图像融合。
- 引入多步去噪采样与特征解耦,显著降低姿态、光照差异带来的伪影。
- 通过掩码打乱技术实现通用换脸模型,可处理发型和饰品替换。
尽管人脸替换任务取得进展,但真实感仍不足,尤其在高姿态变化、色彩差异和遮挡场景下易产生伪影。为此,本文提出一种新方法,更好利用扩散模型解决该问题:(a) 将换脸重构为自监督的训练时修复问题,增强身份传递与目标图像融合;(b) 在训练中引入多步去噪扩散隐式模型(DDIM)采样,强化身份与感知相似性;(c) 引入CLIP特征解耦,从目标图像中提取姿态、表情和光照信息,提升生成真实度;(d) 提出训练时掩码打乱技术,构建通用换脸模型,支持头像级替换,可换发丝与配饰。相比依赖多个现成模型的方法,本方法更统一且鲁棒。在FFHQ和CelebA数据集上的大量实验验证了其有效性与稳健性,实现高保真、低延迟推理。代码已开源:https://github.com/Sanoojan/REFace。
原文摘要 · Abstract (English)
Despite promising progress in face swapping task, realistic swapped images remain elusive, often marred by artifacts, particularly in scenarios involving high pose variation, color differences, and occlusion. To address these issues, we propose a novel approach that better harnesses diffusion models for face-swapping by making following core contributions. (a) We propose to re-frame the face-swapping task as a self-supervised, train-time inpainting problem, enhancing the identity transfer while blending with the target image. (b) We introduce a multi-step Denoising Diffusion Implicit Model (DDIM) sampling during training, reinforcing identity and perceptual similarities. (c) Third, we introduce CLIP feature disentanglement to extract pose, expression, and lighting information from the target image, improving fidelity. (d) Further, we introduce a mask shuffling technique during inpainting training, which allows us to create a so-called universal model for swapping, with an additional feature of head swapping. Ours can swap hair and even accessories, beyond traditional face swapping. Unlike prior works reliant on multiple off-the-shelf models, ours is a relatively unified approach and so it is resilient to errors in other off-the-shelf models. Extensive experiments on FFHQ and CelebA datasets validate the efficacy and robustness of our approach, showcasing high-fidelity, realistic face-swapping with minimal inference time. Our code is available at https://github.com/Sanoojan/REFace.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。