用交错文本语音训练模型，实现零样本流式语音合成。
Interleaved Speech-Text Language Models for Simple Streaming Text-to-Speech Synthesis
- 直接用固定比例的文本与语音标记交错训练。
- 流式合成性能接近非流式模型，差距很小。
- 适合实时接入大语言模型输出的场景。
本文提出用于零样本流式文本到语音合成的交错语音-文本语言模型（IST-LM）。与以往方法不同，IST-LM 直接在固定比例的文本与语音标记交错序列上进行训练，无需强制对齐或复杂设计。文本块与语音块的比例对模型性能至关重要。通过统计分析训练数据并进行相关性研究，发现三个关键因素：1）语音标记与其对应文本标记之间的距离；2）每个语音标记可访问的未来文本标记数量；3）语音标记在其对应文本标记前出现的频率。实验表明，通过合理设置这些因素，可实现性能接近非流式模型的流式TTS系统。IST-LM概念简单、实证有效，以极小开销实现高质量流式语音合成，具备与大语言模型实时文本流集成的广泛潜力。
原文摘要 · Abstract (English)
This paper introduces Interleaved Speech-Text Language Model (IST-LM) for zero-shot streaming Text-to-Speech (TTS). Unlike many previous approaches, IST-LM is directly trained on interleaved sequences of text and speech tokens with a fixed ratio, eliminating the need for additional efforts like forced alignment or complex designs. The ratio of text chunk size to speech chunk size is crucial for the performance of IST-LM. To explore this, we conducted a comprehensive series of statistical analyses on the training data and performed correlation analysis with the final performance, uncovering several key factors: 1) the distance between speech tokens and their corresponding text tokens, 2) the number of future text tokens accessible to each speech token, and 3) the frequency of speech tokens precedes their corresponding text tokens. Experimental results demonstrate how to achieve an optimal streaming TTS system with a limited performance gap compared to its non-streaming counterpart. IST-LM is conceptually simple and empirically powerful, enabling streaming TTS with minimal overhead while largely preserving performance, and offering broad potential for integration with real-time text streams from large language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。