无需训练即可实时生成自然语音，延迟更低。
Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modeling
- 用转换器实时转写文本为语义标记并获取时长对齐信息
- 结合自回归模型重建梅尔频谱，实现高质量语音生成
- 设计删除<bos>机制，极小化延迟，适合低延迟场景
零样本流式文本转语音是人机交互的重要研究方向。现有方法主要依赖前瞻机制，需依赖未来文本以实现自然语音合成，导致处理延迟较高。为解决此问题，本文提出SMLLE，一种逐帧生成高质量语音的流式框架。SMLLE采用转换器实时将文本转为语义标记，并同步获得时长对齐信息；二者联合输入全自回归（AR）流式模型以重构梅尔频谱。为进一步稳定生成过程，设计删除< Bos >机制，使自回归模型能以最小延迟访问未来文本。实验表明，SMLLE优于当前主流流式TTS方法，在句级性能上可媲美非流式系统。演示样本见shy-98.github.io/SMLLE_demo_page/
原文摘要 · Abstract (English)
Zero-shot streaming text-to-speech is an important research topic in human-computer interaction. Existing methods primarily use a lookahead mechanism, relying on future text to achieve natural streaming speech synthesis, which introduces high processing latency. To address this issue, we propose SMLLE, a streaming framework for generating high-quality speech frame-by-frame. SMLLE employs a Transducer to convert text into semantic tokens in real time while simultaneously obtaining duration alignment information. The combined outputs are then fed into a fully autoregressive (AR) streaming model to reconstruct mel-spectrograms. To further stabilize the generation process, we design a Delete < Bos > Mechanism that allows the AR model to access future text introducing as minimal delay as possible. Experimental results suggest that the SMLLE outperforms current streaming TTS methods and achieves comparable performance over sentence-level TTS systems. Samples are available on shy-98.github.io/SMLLE_demo_page/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。