无需语音转录即可跨语言克隆声音,支持14种语言零样本语音合成。
Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder

- 用可学习的说话人编码器从自监督语音中提取音色特征,实现无转录参考克隆。
- 在跨语言评测中平均字错误率仅3.73%,人类评估排名领先开源与商用系统。
- 适合需要多语言语音克隆但无转录文本的场景,如真实环境语音处理。
零样本文本到语音(TTS)技术虽显著提升了语音质量和声音克隆精度,但多数系统仍需推理时依赖参考音频的转录文本,限制了跨语言克隆的应用。本文提出Confucius4-TTS,一个支持14种语言的多语言零样本TTS系统,可在无需音频转录的情况下完成同语种和跨语种参考克隆。该系统采用两阶段架构:基于大语言模型的文本到语义(T2S)模块使用可学习说话人编码器从自监督语音表示中提取音色特征;条件流匹配的语义到声学(S2A)模块将预测的语义令牌转换为梅尔频谱图。同一模型在有参考转录时也可支持续写克隆。模型在大规模多语言语音数据上训练,在公开基准上表现出高可懂度与说话人相似性。在CV3-Eval跨语言基准上,六种方向平均字错误率为3.73%。在内部跨语言测试集上,人类评估整体排名优于近期开源与商业系统。代码、模型检查点及演示已发布于https://github.com/netease-youdao/Confucius4-TTS。
原文摘要 · Abstract (English)
Recent advances in zero-shot text-to-speech (TTS) have substantially improved speech quality and voice cloning fidelity. However, many zero-shot TTS systems still depend on audio prompt transcripts at inference time. This dependency limits cross-lingual voice cloning, since in-the-wild reference audio is often untranscribed. In this technical report, we present Confucius4-TTS, a multilingual zero-shot TTS system that supports 14 languages and performs both intra-lingual and cross-lingual reference cloning without requiring transcripts of audio prompts. Confucius4-TTS follows a two-stage architecture, consisting of text-to-semantic (T2S) and semantic-to-acoustic (S2A) modules. The LLM-based T2S module uses a learnable speaker encoder to extract timbre features from self-supervised speech representations, and the conditional flow-matching S2A module converts the predicted semantic tokens into mel-spectrograms. The same model also supports continuation cloning when a reference transcript is available. Confucius4-TTS is trained on large-scale multilingual speech data. It achieves high intelligibility and speaker similarity on public benchmarks. On the CV3-Eval cross-lingual benchmark, Confucius4-TTS obtains an average WER of 3.73% across six directions. On our internal cross-lingual set, it achieves the best average overall rank in human evaluation among recent open-source and commercial systems. We release code, model checkpoints, and demos at https://github.com/netease-youdao/Confucius4-TTS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。