让语音模型精准控制情绪,且不破坏说话人声音特征。
DUET: Unified Dual-Space Emotion Control for Diffusion and Flow-Matching Driven Text-to-Speech

- 发现情绪信息可线性解码,与说话人特征几乎正交,便于分离控制。
- 统一双空间调节:隐藏层调整情绪方向,频谱层优化声音细节。
- 无需重新训练,适配多种语音模型,适合机器人等具身智能体使用。
基于扩散模型和流匹配的文语合成(TTS)模型在自然度上表现优异,但缺乏显式的情绪控制能力,因为情绪信号与说话人身份混杂在一起。我们发现,情绪嵌入在冻结的隐藏状态中表现为一个可线性解码的方向,几乎与表示说话人身份的方向正交。这一发现启发我们提出一种即插即用的框架 DUET,用于对预训练的扩散模型和流匹配驱动的 TTS 模型进行情绪控制。生成过程中,DUET 通过单步更新实现双空间控制:隐藏空间引导使生成沿目标情绪方向偏移,而梅尔频谱空间的梯度指导则通过可微声码器反向传播,精调频谱细节。我们在三个数据集上对五种架构各异的预训练 TTS 主干网络验证了 DUET,其性能超越 10 种监督类最先进的情感 TTS 基线,在人类评估中达到最高情绪恰当性得分。为进一步展示其定性效果,我们将 DUET 部署于 Ameca 人形机器人,成功生成富有表现力的情感语音,展现出其在具身智能体中实现即插即用情感交互的强大潜力。
原文摘要 · Abstract (English)
Diffusion and flow-matching based text-to-speech (TTS) models excel in naturalness but often lack explicit emotion control, as emotional signals remain entangled with speaker identity. We discover that emotion embedding emerges as a linearly decodable direction of frozen hidden states, nearly orthogonal to the direction embedding speaker identity. This inspires a plug-and-play framework DUET for emotion control over pretrained diffusion and flow-matching based TTS models. During generation, DUET unifies dual-space control to achieve fine-grained emotion intervention in a single per-step update: hidden space steering shifts generation along the target emotion direction, while mel-space guidance refines spectral details through gradients backpropagated from a differentiable vocoder. We validate DUET on five architecturally diverse pretrained TTS backbones across three datasets, where it outperforms 10 supervised state-of-the-art emotional TTS baselines across paradigms and achieves the highest human-rated emotion appropriateness. To further showcase its qualitative behavior, we deploy DUET on an Ameca humanoid robot, where it produces richly expressive emotional speech on the humanoid, demonstrating the strong potential for plug-and-play affective interaction for embodied agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。