实现多人实时对话的自然人脸生成,支持任意人数与流畅互动。
Towards Flexible, Natural, Efficient Interaction for Conversational Talking Face Generation

- 基于运动建模架构,显式捕捉多轮对话动态,支持任意人数
- 30帧/秒实时生成,通过分量解耦提升唇同步与眨眼真实感
- 新增多人群体对话数据集,含3D人脸增强,适合交互系统研发
对话式说话人脸生成近年来备受关注,旨在合成角色能动态交谈、倾听并回应的交互视频。该任务面临三大挑战:灵活性(支持任意人数、多轮对话)、自然性(保持动作连贯与非语言反馈)和效率(实现实时生成与低计算开销)。现有方法难以兼顾三者。为此,我们提出InterTalk框架,基于运动建模实现高效交互式说话人脸生成。通过显式建模每位参与者的多轮对话动态,突破人数限制;引入多参与者运动反馈与迭代生成策略,提升行为自然性;将运动解耦为多个面部成分,实现唇同步与眨眼等细节的精准优化。此外,我们构建新多人群体对话数据集,并加入3D人脸数据增强。大量实验表明,InterTalk在保持30 FPS实时性能的同时,显著提升交互质量。
原文摘要 · Abstract (English)
Conversational talking face generation has recently attracted increasing attention, aiming to synthesize interactive talking videos where characters speak, listen, and respond dynamically to each other. This task presents three core challenges: 1) Flexibility: enabling multi-round dialogues with an arbitrary number of participants; 2) Naturalness: maintaining coherent motion and appropriate non-verbal feedback throughout the interaction; and 3) Efficiency: achieving real-time generation and low computation overhead for long-term continuous online conversation. Despite recent advances, existing methods still fall short in balancing all three requirements. To bridge this gap, we introduce InterTalk, a novel and efficient framework designed for highly interactive conversational talking face generation. Built upon a motion-based architecture, InterTalk supports real-time conversation synthesis. Our method achieves strong flexibility by explicitly modeling multi-round conversational dynamics among each participant, eliminating constraints on their numbers. To enhance interactivity, we incorporate motion feedback from multiple participants and introduce an iterative generation strategy for more natural behaviors. Besides, we disentangle motion into several facial components, enabling targeted refinements for natural response such as precise lip sync and realistic eye blinking. Finally, we construct a new multi-person conversational dataset and enrich it with 3D face-based data augmentation. Extensive experiments demonstrate that InterTalk achieves superior interaction quality while maintaining real-time performance at 30 FPS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。