S5-TTS实现低延迟语音合成，接收首个词即可开始生成。
Streaming T5-based Text-to-Speech Synthesis with Limited Lookahead
- 采用编码器-解码器架构与单调对齐学习，实现逐词增量生成。
- 在有限前瞻下保持语音可懂度与说话人相似性，端到端延迟显著降低。
- 适合需要实时响应的对话AI系统，支持零样本语音克隆。
在级联大语言模型-语音合成系统中，流式语音合成仍面临延迟挑战，因多数语音合成模型需完整上下文才能启动生成。本文提出S5-TTS，一种T5-TTS的流式变体，通过编码器-解码器语言建模与单调对齐学习，实现低延迟、逐词增量语音合成。S5-TTS在接收到前几个词后即可立即开始生成，大幅降低端到端响应延迟。为在有限前瞻条件下维持质量，引入基于卷积的辅助注意力的前瞻因果掩码机制，保障语音可懂度与说话人相似性，并采用交错多源蒸馏进一步恢复自然度。实验表明，S5-TTS在质量上可媲美全上下文T5-TTS，支持高说话人相似性的零样本合成，显著降低实际对话AI系统的端到端延迟。
原文摘要 · Abstract (English)
Streaming text-to-speech synthesis in cascaded LLM-TTS systems still faces latency challenges as most TTS models require full context before initiating generation. We present S5-TTS, a streaming variant of T5-TTS that enables low-latency, word-by-word incremental speech synthesis through encoder-decoder language modeling and monotonic alignment learning. S5-TTS begins generating speech immediately after receiving the first few words, substantially reducing end-to-end response latency. To maintain quality under limited lookahead, we introduce a lookahead-causal masking mechanism with Conv-based auxiliary attention that preserves intelligibility and speaker similarity, and employ interleaved multi-source distillation to further restore naturalness. Experiments show that S5-TTS achieves comparable quality to full-context T5-TTS, supports zero-shot synthesis with high speaker similarity, and significantly reduces end-to-end latency for practical conversational AI systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。