让语音合成实时响应，实现零延迟对话。
SpeakStream: Streaming Text-to-Speech with Interleaved Data
- 用交错文本-语音数据训练解码器架构，支持边输入边生成。
- 首音延迟显著降低，优于现有非流式合成系统。
- 适合需要快速响应的对话智能体场景。
传统文本转语音（TTS）系统因需完整语句输入而存在延迟瓶颈，严重制约了流式大语言模型在对话AI中的应用。这些系统通常以完整语句为单位进行训练和推理，即使优化推理速度，与流式输出的大语言模型结合时仍引入不可接受的延迟，尤其在需要低首词延迟的对话场景中尤为突出。本文提出SpeakStream，一种基于解码器架构的流式TTS系统，可从流式文本中增量生成音频。该系统采用交错文本-语音数据进行下一帧预测损失训练。推理时，能边接收文本流边生成语音，特别适用于大语言模型向语音合成系统流式输出的级联对话智能体。实验表明，SpeakStream在首词延迟方面达到当前最优水平，同时保持非流式TTS系统的音质水准。
原文摘要 · Abstract (English)
The latency bottleneck of traditional text-to-speech (TTS) systems fundamentally hinders the potential of streaming large language models (LLMs) in conversational AI. These TTS systems, typically trained and inferenced on complete utterances, introduce unacceptable delays, even with optimized inference speeds, when coupled with streaming LLM outputs. This is particularly problematic for creating responsive conversational agents where low first-token latency is critical. In this paper, we present SpeakStream, a streaming TTS system that generates audio incrementally from streaming text using a decoder-only architecture. SpeakStream is trained using a next-step prediction loss on interleaved text-speech data. During inference, it generates speech incrementally while absorbing streaming input text, making it particularly suitable for cascaded conversational AI agents where an LLM streams text to a TTS system. Our experiments demonstrate that SpeakStream achieves state-of-the-art latency results in terms of first-token latency while maintaining the quality of non-streaming TTS systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。