FlashTTS实现325ms首包延迟,支持实时语音对话。
FlashTTS: Fast Streaming TTS with MTP Acceleration and X-pred Mean Flow Distillation
- 采用延迟多轨架构,原生支持流式输入输出,无需句子级缓存。
- 结合MTP与X-pred均值流匹配,仅2次函数求值即完成音素到声谱生成。
- 开源模型可零样本克隆声音,适合实时语音交互系统部署。
近期语音对话系统的发展要求文本转语音(TTS)模型具备更快响应速度和更低延迟。现代语音对话系统对TTS模型提出两大核心需求:低延迟与支持流式输入输出。然而,现有基于单码本大语言模型的TTS方法大多依赖多阶段流水线,缺乏原生流式能力,通常因自回归预测缓慢和多步流匹配导致端到端延迟过高。为此,我们提出FlashTTS——一个开源、低延迟的流式TTS框架。FlashTTS引入延迟多轨架构,原生处理流式文本与语音输入,消除句子级缓冲需求。为加速声学生成,融合并行多标记预测(MTP)与X-pred均值流匹配解码器,仅需两次函数求值(2-NFE)即可实现高保真音素到声谱转换。通过联合优化输入处理与解码效率,FlashTTS为实时语音对话系统提供实用基础。实验表明,相比稳健的流式基线,FlashTTS将首包延迟显著降低至325ms,同时保持强大的零样本语音克隆与跨语言可理解性。语音样例已提供,模型代码与检查点将开源。
原文摘要 · Abstract (English)
Recent progress in speech dialogue systems requires Text-to-Speech (TTS) models to be faster and more responsive. Modern speech dialogue systems impose two primary requirements on TTS models: low latency and support for streaming inputs and outputs. However, most existing single-codebook LLM-based TTS methods rely on multi-stage pipelines that lack native streaming capabilities. These systems typically suffer from high end-to-end latency due to slow autoregressive prediction and multi-step flow matching. To address these limitations, we propose FlashTTS, an open-source and low-latency streaming TTS framework. FlashTTS introduces a lagged multi-track architecture that natively processes streaming text and speech inputs, thereby eliminating the need for sentence-level buffering. To accelerate acoustic generation, we integrate parallel Multi-Token Prediction (MTP) with an X-pred mean flow matching decoder. This configuration achieves high-fidelity token-to-mel generation in exactly two function evaluations (2-NFE). By jointly optimizing input processing and decoding efficiency, FlashTTS offers a practical foundation for real-time speech dialogue systems. Experiments show that FlashTTS substantially reduces First-Packet Latency to 325ms compared to robust streaming baselines, all while preserving strong zero-shot voice cloning and cross-lingual intelligibility. Speech samples are available. The model code and checkpoints will be released as open source.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。