arXiv:2409.15179cs.CV2024-09被引 2

通过双模态特征调制,提升人脸动画的稳定性和细节保真度。

MIMAFace: Face Animation via Motion-Identity Modulated Appearance Feature Learning

  • 用运动与身份双层调制模块,精准控制面部表情与注视方向
  • 在公开数据集上实现高质量、跨片段时序一致的动画生成
  • 支持多种驱动源输入,适合影视特效与虚拟人开发

当前基于扩散模型的人脸动画方法通常依赖参考网络(ReferenceNet,即U-Net副本)和大量自采集数据来学习外观特征,因为鲁棒的外观特征对保证时序稳定性至关重要。然而,在公共数据集上训练时,图像质量与时序一致性常存在明显差距。为此,我们深入分析了人脸动画任务中的关键外观特征,包括与运动无关的(如服装、背景)和与运动相关的(如面部细节)纹理成分,以及高层判别性身份特征。基于此,我们提出运动-身份调制外观学习模块(MIA),在运动和身份两个层面调制CLIP特征。此外,为解决片段间的语义/颜色不连续问题,设计了跨片段亲和力学习模块(ICA),以建模片段间的时序关系。本方法实现了精确的面部运动控制(如表情与注视)、忠实的身份保留,并生成兼具片段内/间时序一致性的动画视频。同时,该方法可轻松适配多种驱动源模态。大量实验验证了其优越性。

原文摘要 · Abstract (English)

Current diffusion-based face animation methods generally adopt a ReferenceNet (a copy of U-Net) and a large amount of curated self-acquired data to learn appearance features, as robust appearance features are vital for ensuring temporal stability. However, when trained on public datasets, the results often exhibit a noticeable performance gap in image quality and temporal consistency. To address this issue, we meticulously examine the essential appearance features in the facial animation tasks, which include motion-agnostic (e.g., clothing, background) and motion-related (e.g., facial details) texture components, along with high-level discriminative identity features. Drawing from this analysis, we introduce a Motion-Identity Modulated Appearance Learning Module (MIA) that modulates CLIP features at both motion and identity levels. Additionally, to tackle the semantic/ color discontinuities between clips, we design an Inter-clip Affinity Learning Module (ICA) to model temporal relationships across clips. Our method achieves precise facial motion control (i.e., expressions and gaze), faithful identity preservation, and generates animation videos that maintain both intra/inter-clip temporal consistency. Moreover, it easily adapts to various modalities of driving sources. Extensive experiments demonstrate the superiority of our method.

人脸动画扩散模型时序一致性CLIP

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。