用ControlNet实现细粒度情绪控制的语音合成，无需重训练。
TTS-CtrlNet: Time varying emotion aligned text-to-speech generation with ControlNet
- 通过可训练副本在不改原模型前提下加入情绪条件控制
- 在预训练大模型上实现时变情绪控制，保持自然度与零样本克隆能力
- 提供可扩展的情绪控制方案，适合需灵活调节情绪的语音应用
近期文本到语音（TTS）技术虽已实现自然语音合成，但细粒度、时变情绪控制仍具挑战。现有方法多仅支持整句级控制，且需大规模情感语音数据微调模型，易导致性能下降。受ControlNet启发，我们提出首个基于ControlNet的可控流匹配TTS方法（TTS-CtrlNet），冻结原始模型并引入可训练副本处理额外条件。实验表明，TTS-CtrlNet可在不改变原模型的情况下，有效添加直观、可扩展的时变情绪控制，同时保留原始模型能力（如零样本语音克隆与自然度）。我们还提供了实用方案：1）最优架构设计与模块分析；2）特定情绪的流步数调整；3）灵活控制尺度。在情感相似度指标（Emo-SIM与Aro-Val SIM）上达到当前最优性能。
原文摘要 · Abstract (English)
Recent advances in text-to-speech (TTS) have enabled natural speech synthesis, but fine-grained, time-varying emotion control remains challenging. Existing methods often allow only utterance-level control and require full model fine-tuning with a large emotion speech dataset, which can degrade performance. Inspired by adding conditional control to the existing model in ControlNet (Zhang et al, 2023), we propose the first ControlNet-based approach for controllable flow-matching TTS (TTS-CtrlNet), which freezes the original model and introduces a trainable copy of it to process additional conditions. We show that TTS-CtrlNet can boost the pretrained large TTS model by adding intuitive, scalable, and time-varying emotion control while inheriting the ability of the original model (e.g., zero-shot voice cloning & naturalness). Furthermore, we provide practical recipes for adding emotion control: 1) optimal architecture design choice with block analysis, 2) emotion-specific flow step, and 3) flexible control scale. Experiments show that ours can effectively add an emotion controller to existing TTS, and achieves state-of-the-art performance with emotion similarity scores: Emo-SIM and Aro-Val SIM. The project page is available at: https://curryjung.github.io/ttsctrlnet_project_page
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。