让虚拟人物会情绪地对话,自动切换说听状态
Warm Chat: Diffuse Emotion-aware Interactive Talking Head Avatar with Tree-Structured Guidance
- 用树状结构管理对话状态,实时追踪情绪变化
- 生成长时间一致的口型动作,支持双向自然互动
- 适合需要情感化交互的虚拟客服、数字人场景
生成模型已实现令人惊叹的说话头像生成,使人工智能更具生命感。然而,现有方法大多仅支持单向图像动画,少数支持双向对话的模型缺乏精确的情绪自适应能力,严重限制了实际应用。本文提出 Warm Chat,一种面向二人互动的新型情绪感知说话头像生成框架。利用大语言模型(如 GPT-4)的对话生成能力,该方法生成在时序上一致、具有丰富情绪变化的虚拟形象,并能无缝切换说话与倾听状态。我们设计了一个基于 Transformer 的头像掩码生成器,在潜在掩码空间中学习时序一致的动作特征,可生成任意长度且时序一致的掩码序列以约束头部运动。此外,引入交互式说话树结构表示对话状态转移,每个节点包含父子兄弟关系及当前角色情绪状态。通过反向层级遍历,从当前节点提取丰富的历史情绪线索,用于表情合成。大量实验表明该方法性能优越且有效。
原文摘要 · Abstract (English)
Generative models have advanced rapidly, enabling impressive talking head generation that brings AI to life. However, most existing methods focus solely on one-way portrait animation. Even the few that support bidirectional conversational interactions lack precise emotion-adaptive capabilities, significantly limiting their practical applicability. In this paper, we propose Warm Chat, a novel emotion-aware talking head generation framework for dyadic interactions. Leveraging the dialogue generation capability of large language models (LLMs, e.g., GPT-4), our method produces temporally consistent virtual avatars with rich emotional variations that seamlessly transition between speaking and listening states. Specifically, we design a Transformer-based head mask generator that learns temporally consistent motion features in a latent mask space, capable of generating arbitrary-length, temporally consistent mask sequences to constrain head motions. Furthermore, we introduce an interactive talking tree structure to represent dialogue state transitions, where each tree node contains information such as child/parent/sibling nodes and the current character's emotional state. By performing reverse-level traversal, we extract rich historical emotional cues from the current node to guide expression synthesis. Extensive experiments demonstrate the superior performance and effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。