通过控制面部动作单元，生成更真实有情绪的虚拟说话头视频。
AUHead: Realistic Emotional Talking Head Generation via Action Units Control
- 分两阶段：先用大模型解析语音中的细微表情动作单元，再用扩散模型生成视频。
- 在基准数据集上情感真实度和口型同步性显著优于现有方法。
- 适合需要精细情绪控制的虚拟人、影视制作与交互系统开发者。
真实感说话头视频生成对虚拟形象、影视制作和交互系统至关重要。当前方法因缺乏细粒度情绪控制而难以呈现自然情感表达。为此，我们提出一种两阶段新方法 AUHead，将面部动作单元（AUs）从音频中解耦并实现可控生成。第一阶段利用大音频-语言模型（ALMs），通过时空动作单元标记和“情绪→动作单元”链式思维机制，有效捕捉语音中的细微情感线索。第二阶段设计一种基于动作单元驱动的可控扩散模型，将AU序列映射为结构化二维人脸表示以提升空间保真度，并在交叉注意力模块中建模AU与视觉的交互。为实现灵活的AU质量权衡，引入推理阶段的解耦引导策略，进一步增强情感表现力与身份一致性。在多个基准数据集上的结果表明，该方法在情感真实度、口型同步性和视觉连贯性方面均达到领先水平，显著超越现有技术。代码已开源：https://github.com/laura990501/AUHead_ICLR。
原文摘要 · Abstract (English)
Realistic talking-head video generation is critical for virtual avatars, film production, and interactive systems. Current methods struggle with nuanced emotional expressions due to the lack of fine-grained emotion control. To address this issue, we introduce a novel two-stage method (AUHead) to disentangle fine-grained emotion control, i.e. , Action Units (AUs), from audio and achieve controllable generation. In the first stage, we explore the AU generation abilities of large audio-language models (ALMs), by spatial-temporal AU tokenization and an "emotion-then-AU" chain-of-thought mechanism. It aims to disentangle AUs from raw speech, effectively capturing subtle emotional cues. In the second stage, we propose an AU-driven controllable diffusion model that synthesizes realistic talking-head videos conditioned on AU sequences. Specifically, we first map the AU sequences into the structured 2D facial representation to enhance spatial fidelity, and then model the AU-vision interaction within cross-attention modules. To achieve flexible AU-quality trade-off control, we introduce an AU disentanglement guidance strategy during inference, further refining the emotional expressiveness and identity consistency of the generated videos. Results on benchmark datasets demonstrate that our approach achieves competitive performance in emotional realism, accurate lip synchronization, and visual coherence, significantly surpassing existing techniques. Our implementation is available at https://github.com/laura990501/AUHead_ICLR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。