用大模型直接生成多语言语音，省去传统转换步骤。
Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis
- 用大模型提取语言特征，跳过繁琐的音素转换
- 支持复杂语境和多语言语音合成，效果优于基线模型
- 开源框架，适合语音克隆和智能交互场景
文本转语音系统在处理复杂语言特征、多音字表达和自然多语言语音方面仍面临挑战，这些能力对未来的AI应用至关重要。本文提出Fish-Speech，一种基于串行快慢双重自回归（Dual-AR）架构的新框架，以增强分组有限标量向量量化（GFSQ）在序列生成任务中的稳定性。该架构提升了代码本处理效率，同时保持高保真输出，特别适用于人机交互与语音克隆。Fish-Speech利用大语言模型（LLMs）进行语言特征提取，无需传统的音素转换（G2P），从而简化合成流程并增强多语言支持。此外，我们通过GFSQ构建了FF-GAN，实现更优压缩比与接近100%的代码本利用率。实验表明，Fish-Speech在复杂语言场景和语音克隆任务中显著优于基线模型，展现出推动TTS技术发展的潜力。实现代码已开源：https://github.com/fishaudio/fish-speech。
原文摘要 · Abstract (English)
Text-to-Speech (TTS) systems face ongoing challenges in processing complex linguistic features, handling polyphonic expressions, and producing natural-sounding multilingual speech - capabilities that are crucial for future AI applications. In this paper, we present Fish-Speech, a novel framework that implements a serial fast-slow Dual Autoregressive (Dual-AR) architecture to enhance the stability of Grouped Finite Scalar Vector Quantization (GFSQ) in sequence generation tasks. This architecture improves codebook processing efficiency while maintaining high-fidelity outputs, making it particularly effective for AI interactions and voice cloning. Fish-Speech leverages Large Language Models (LLMs) for linguistic feature extraction, eliminating the need for traditional grapheme-to-phoneme (G2P) conversion and thereby streamlining the synthesis pipeline and enhancing multilingual support. Additionally, we developed FF-GAN through GFSQ to achieve superior compression ratios and near 100\% codebook utilization. Our approach addresses key limitations of current TTS systems while providing a foundation for more sophisticated, context-aware speech synthesis. Experimental results show that Fish-Speech significantly outperforms baseline models in handling complex linguistic scenarios and voice cloning tasks, demonstrating its potential to advance TTS technology in AI applications. The implementation is open source at \href{https://github.com/fishaudio/fish-speech}{https://github.com/fishaudio/fish-speech}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。