通过控制面部动作单元，实现高保真人脸表情精细编辑。
MagicFace: High-Fidelity Facial Expression Editing with Action-Unit Control
- 基于扩散模型与动作单元控制，实现连续可解释的表情编辑。
- 在保持身份、姿态和背景一致的前提下，生成高质量表情变化图像。
- 适合需要精确控制人脸表情的影视制作与虚拟人应用。
我们提出一种通过控制同一人面部动作单元（AU）相对变化来实现人脸表情编辑的方法。该方法可在保持身份、姿态、背景及面部细节一致的前提下，实现细粒度、连续且可解释的表情编辑。核心模型MagicFace采用基于AU变化条件的扩散模型，并结合身份编码器以保留高一致性面部特征。为保留输入身份的外观细节，我们利用预训练Stable-Diffusion模型，并设计自注意力机制的身份编码器融合外观特征。为保持背景与姿态一致性，引入高效属性控制器，显式告知模型目标的当前背景与姿态信息。通过将AU变化注入去噪UNet，模型可对任意身份生成多种AU组合的表情动画，在高保真表情编辑方面优于现有方法。代码已公开于https://github.com/weimengting/MagicFace。
原文摘要 · Abstract (English)
We address the problem of facial expression editing by controling the relative variation of facial action-unit (AU) from the same person. This enables us to edit this specific person's expression in a fine-grained, continuous and interpretable manner, while preserving their identity, pose, background and detailed facial attributes. Key to our model, which we dub MagicFace, is a diffusion model conditioned on AU variations and an ID encoder to preserve facial details of high consistency. Specifically, to preserve the facial details with the input identity, we leverage the power of pretrained Stable-Diffusion models and design an ID encoder to merge appearance features through self-attention. To keep background and pose consistency, we introduce an efficient Attribute Controller by explicitly informing the model of current background and pose of the target. By injecting AU variations into a denoising UNet, our model can animate arbitrary identities with various AU combinations, yielding superior results in high-fidelity expression editing compared to other facial expression editing works. Code is publicly available at https://github.com/weimengting/MagicFace.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。