IndexTTS实现高效零样本语音合成，中文发音更可控。
IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System
- 融合字符与拼音的混合建模，提升多音字和生僻字发音控制力。
- 相比XTTS，自然度、内容一致性与克隆稳定性显著提升。
- 训练简单、推理快，适合工业级语音合成应用。
近期基于大语言模型的文本到语音（TTS）系统因高自然度和强大的零样本语音克隆能力，逐渐成为行业主流。本文介绍的IndexTTS系统以XTTS和Tortoise模型为基础，引入多项创新改进。在中文场景中，采用字符与拼音混合建模方法，实现对多音字和长尾字发音的可控生成；对比分析了向量量化（VQ）与有限标量量化（FSQ）在声学语音标记码本使用上的效果；为增强语音克隆的效果与稳定性，引入基于Conformer的语音条件编码器，并用BigVGAN2替代原语音解码器。相较于XTTS，IndexTTS在自然度、内容一致性和零样本语音克隆方面均有显著提升。与Fish-Speech、CosyVoice2、FireRedTTS及F5-TTS等开源主流系统相比，IndexTTS具备更简单的训练流程、更强的可控性与更快的推理速度，性能也更优。演示地址见https://index-tts.github.io。
原文摘要 · Abstract (English)
Recently, large language model (LLM) based text-to-speech (TTS) systems have gradually become the mainstream in the industry due to their high naturalness and powerful zero-shot voice cloning capabilities.Here, we introduce the IndexTTS system, which is mainly based on the XTTS and Tortoise model. We add some novel improvements. Specifically, in Chinese scenarios, we adopt a hybrid modeling method that combines characters and pinyin, making the pronunciations of polyphonic characters and long-tail characters controllable. We also performed a comparative analysis of the Vector Quantization (VQ) with Finite-Scalar Quantization (FSQ) for codebook utilization of acoustic speech tokens. To further enhance the effect and stability of voice cloning, we introduce a conformer-based speech conditional encoder and replace the speechcode decoder with BigVGAN2. Compared with XTTS, it has achieved significant improvements in naturalness, content consistency, and zero-shot voice cloning. As for the popular TTS systems in the open-source, such as Fish-Speech, CosyVoice2, FireRedTTS and F5-TTS, IndexTTS has a relatively simple training process, more controllable usage, and faster inference speed. Moreover, its performance surpasses that of these systems. Our demos are available at https://index-tts.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。