让语音合成模型实时处理连续文本流，支持无限长语音生成。
Zero-Shot Text-to-Speech from Continuous Text Streams
- 用Mamba架构结合交叉注意力，实现线性时间推理和持续文本处理。
- 通过旋转位置编码滑动窗口，支持无限长度文本流的语音生成。
- 语义引导解码使语音与文本对齐，适合直播、实时对话等场景。
现有零样本语音合成系统通常需完整句子输入，且受限于训练时的最大时长。但在许多流式应用中，文本以短片段连续到达，要求系统即时响应。我们识别出块级流式处理的核心能力，提出LiveSpeech 2：一种面向流式场景的模型，支持无限长语音生成、文本-音频流同步及短语音块间无缝衔接。为此，我们提出：(1) 采用Mamba序列建模架构，其具有线性时间解码优势，并引入交叉注意力机制实现条件控制；(2) 在交叉注意力计算中使用旋转位置编码，使模型能通过滑动窗口处理无限文本流；(3) 采用语义引导解码，在推理阶段以极低开销实现语音与转录文本的对齐。实验表明，该模型在性能上可媲美基于语言模型的前沿零样本TTS系统，同时具备广泛支持流式应用场景的灵活性。
原文摘要 · Abstract (English)
Existing zero-shot text-to-speech (TTS) systems are typically designed to process complete sentences and are constrained by the maximum duration for which they have been trained. However, in many streaming applications, texts arrive continuously in short chunks, necessitating instant responses from the system. We identify the essential capabilities required for chunk-level streaming and introduce LiveSpeech 2, a stream-aware model that supports infinitely long speech generation, text-audio stream synchronization, and seamless transitions between short speech chunks. To achieve these, we propose (1) adopting Mamba, a class of sequence modeling distinguished by linear-time decoding, which is augmented by cross-attention mechanisms for conditioning, (2) utilizing rotary positional embeddings in the computation of cross-attention, enabling the model to process an infinite text stream by sliding a window, and (3) decoding with semantic guidance, a technique that aligns speech with the transcript during inference with minimal overhead. Experimental results demonstrate that our models are competitive with state-of-the-art language model-based zero-shot TTS models, while also providing flexibility to support a wide range of streaming scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。