解决人脸表情与音频情绪冲突问题，生成更准确的情感说话视频。
CEM-Net: Cross-Emotion Memory Network for Emotional Talking Face Generation
- 用交叉重建策略增强音频情绪，减少参考图像情绪干扰。
- 引入情绪桥接记忆模块，补足音频情绪下的面部动作信息。
- 适合需要高情感一致性的虚拟人、影视合成等应用。
情感说话脸生成旨在根据参考图像和驱动音频，生成与内容和情绪匹配的说话视频。然而，现有方法忽视了参考图像可能带有强烈情绪而与音频情绪冲突的问题，导致情感失准和生成结果扭曲。为此，本文提出交叉情绪记忆网络（CEM-Net），在参考图像情绪强烈时仍能生成与驱动音频情绪一致的说话脸。首先设计音频情绪增强模块（AEE），采用交叉重建训练策略强化音频情绪，缓解参考图像情绪的干扰。其次，由于参考图像无法提供音频情绪下的充分面部运动信息，引入情绪桥接记忆模块（EBM），将参考图像情绪表达位移迁移到音频情绪，并存储于记忆中。推理时，以跨情绪特征为查询，可检索到匹配的位移。大量实验表明，所提方法能生成更具表现力、自然且口型同步的说话视频，且情感准确性显著提升。
原文摘要 · Abstract (English)
Emotional talking face generation aims to animate a human face in given reference images and generate a talking video that matches the content and emotion of driving audio. However, existing methods neglect that reference images may have a strong emotion that conflicts with the audio emotion, leading to severe emotion inaccuracy and distorted generated results. To tackle the issue, we introduce a cross-emotion memory network(CEM-Net), designed to generate emotional talking faces aligned with the driving audio when reference images exhibit strong emotion. Specifically, an Audio Emotion Enhancement module(AEE) is first devised with the cross-reconstruction training strategy to enhance audio emotion, overcoming the disruption from reference image emotion. Secondly, since reference images cannot provide sufficient facial motion information of the speaker under audio emotion, an Emotion Bridging Memory module(EBM) is utilized to compensate for the lacked information. It brings in expression displacement from the reference image emotion to the audio emotion and stores it in the memory.Given a cross-emotion feature as a query, the matching displacement can be retrieved at inference time. Extensive experiments have demonstrated that our CEM-Net can synthesize expressive, natural and lip-synced talking face videos with better emotion accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。