让对话语音合成更真实,突破情绪类别限制。
AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesis

- 用离散情绪编码学习更自然的情绪空间,替代固定七类情绪。
- 融合多轮对话中冗余信息,保留关键情感上下文。
- 结合真实情绪与声学先验,生成更具表现力的语音。
对话式语音合成(CSS)旨在实现用户与智能体交互中具有人类情感表达和上下文一致性的语音生成。现有方法受限于预定义情绪标签空间(如七类基本情绪),难以呈现真实情感;同时多轮对话历史中的冗余多模态标记干扰上下文理解。为此,我们提出 AuEmoChat 框架,实现真实情感的理解与渲染。首先,设计 AuEmoCodec,通过有限标量量化从大规模情感语音中学习离散的真实情绪标记空间,提供比传统基础情绪类别更自然的情感表征。其次,提出 AuEmoToMe 算法,引导融合多模态对话历史中的冗余标记,同时保留情感相关上下文。将其集成到自回归文本-语音模型中,用于预测目标真实情绪标记与语音标记。最后,引入真实情感流匹配机制,联合条件于合并后的对话上下文、目标真实情绪和声学先验,实现语音渲染。在 NCSSD-EmCap 数据集上的大量实验表明,AuEmoChat 显著优于现有先进基线,生成更具表现力与真实感的情感语音。代码与语音演示将公开于:https://github.com/AI-S2-Lab/AuEmoChat。
原文摘要 · Abstract (English)
Conversational Speech Synthesis (CSS) aims to synthesize speech with human-like emotional expression and contextual consistency in user-agent interactions. Existing CSS methods struggle to render authentic human emotions due to limited predefined emotion label spaces (e.g., seven emotion categories), while redundant multimodal tokens in multi-turn dialogue history interfere with context understanding. To address these issues, we propose AuEmoChat, a CSS framework for authentic emotion understanding and rendering. First, we develop AuEmoCodec, which learns a discrete authentic emotion token space from large-scale emotional speech via finite scalar quantization, enabling a more authentic emotion representation than limited basic emotion categories. Furthermore, we propose AuEmoToMe, an authentic-emotion-guided token merging algorithm that merges redundant tokens in multimodal dialogue history while preserving emotion-relevant context. We integrate it into an autoregressive text-speech model to predict the target authentic emotion token and speech tokens. Finally, we propose Authentic Emotion Flow Matching, which renders speech by jointly conditioning on merged dialogue context, target authentic emotion, and acoustic priors. Extensive experiments on the NCSSD-EmCap dataset demonstrate that AuEmoChat outperforms state-of-the-art CSS baselines and generates more expressive and authentic emotional speech. The code and speech demos will be available at: https://github.com/AI-S2-Lab/AuEmoChat.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。