分离语音的语义、韵律和音色，实现零样本可控语音生成
DisCo-Speech: Controllable Zero-Shot Speech Generation with A Disentangled Speech Codec
- 通过两阶段设计解耦语音内容、韵律与音色，构建独立控制基础
- 在零样本条件下实现音色克隆与韵律灵活控制，性能优于现有方法
- 适合需要语音风格自由调控的研究与应用，如语音助手、有声书生成
基于编解码器的语言模型（LM）已革新文本到语音（TTS）技术。然而，标准编解码器将音色与韵律混合在一起，阻碍了基于续写机制的模型进行独立控制。为解决此问题，我们提出 DisCo-Speech，一种零样本可控语音生成框架，包含解耦语音编解码器（DisCodec）和基于LM的生成器。核心组件 DisCodec 采用两阶段设计：1）三因子解耦，通过并行编码器与混合损失函数，将语音分离为内容、韵律与音色子空间；2）融合与重构，将内容与韵律合并为统一的内容-韵律标记以供LM预测，同时联合优化重建效果，缓解解耦与重建之间的权衡。该设计使语言模型可基于风格提示进行韵律续写，而解码器注入目标音色，实现灵活的零样本控制。实验表明，DisCo-Speech 在语音克隆方面表现具有竞争力，在零样本韵律控制上更优。通过在编解码器层面解决核心耦合问题，DisCo-Speech 为可控语音合成提供了稳健基础。
原文摘要 · Abstract (English)
Codec-based language models (LMs) have revolutionized text-to-speech (TTS). However, standard codecs entangle timbre and prosody, which hinders independent control in continuation-based LMs. To tackle this challenge, we propose DisCo-Speech, a zero-shot controllable TTS framework featuring a disentangled speech codec (DisCodec) and an LM-based generator. The core component DisCodec employs a two-stage design: 1) tri-factor disentanglement to separate speech into content, prosody, and timbre subspaces via parallel encoders and hybrid losses; and 2) fusion and reconstruction that merges content and prosody into unified content-prosody tokens suitable for LM prediction, while jointly optimizing reconstruction to address the disentanglement-reconstruction trade-off. This allows the LM to perform prosodic continuation from a style prompt while the decoder injects target timbre, enabling flexible zero-shot control. Experiments demonstrate that DisCo-Speech achieves competitive voice cloning and superior zero-shot prosody control. By resolving the core entanglement at the codec level, DisCo-Speech provides a robust foundation for controllable speech synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。