让语音合成模型学会在流式输入时提前停顿，避免长文本失真。
Prosodic Boundary-Aware Streaming Generation for LLM-Based TTS with Streaming Text Input
- 用弱时间对齐数据微调大模型，使其在有限未来文本下学会在语义边界提前停止
- 长文本合成时字错误率从71.0%降至4.8%，下降66.2个百分点
- 适合需要实时交互的语音系统，尤其擅长处理连续输入的长文本
流式文本输入的语音合成对交互系统至关重要，但面临两大挑战：因缺乏前瞻信息导致韵律不自然，以及因上下文无限增长引发的长文本崩溃。本文提出一种基于韵律边界的后训练策略，利用弱时间对齐数据微调预训练的大型语言模型（LLM）基语音合成模型。具体地，模型被训练为在仅获得有限未来文本时，能在指定内容边界处提前停止。推理阶段采用滑动窗口提示，传递先前文本与语音标记，确保上下文有界且合成结果无缝衔接。实验表明，该方法在短文本和长文本场景下均优于CosyVoice-Style交错基线。尤其在长文本合成中，字错误率绝对降低66.2%（从71.0%降至4.8%），说话人和情感相似度分别相对提升16.1%和1.5%，为增量式流式语音合成提供了稳健解决方案。
原文摘要 · Abstract (English)
Streaming TTS that receives streaming text is essential for interactive systems, yet this scheme faces two major challenges: unnatural prosody due to missing lookahead and long-form collapse due to unbounded context. We propose a prosodic-boundary-aware post-training strategy, adapting a pretrained LLM-based TTS model using weakly time-aligned data. Specifically, the model is adapted to learn early stopping at specified content boundaries when provided with limited future text. During inference, a sliding-window prompt carries forward previous text and speech tokens, ensuring bounded context and seamless concatenation. Evaluations show our method outperforms CosyVoice-Style interleaved baseline in both short and long-form scenarios. In long-text synthesis, especially, it achieves a 66.2% absolute reduction in word error rate (from 71.0% to 4.8%) and increases speaker and emotion similarity by 16.1% and 1.5% relatively, offering a robust solution for streaming TTS with incremental text.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。