arXiv:2502.11094cs.SDcs.AI2025-02被引 1

提出SyncSpeech,实现高效低延迟语音合成。

SyncSpeech: Efficient and Low-Latency Text-to-Speech based on Temporal Masked Transformer

  • 用时序掩码变换器统一自回归与非自回归优势。
  • 首包延迟降低5.8倍,实时因子提升8.8倍。
  • 适合流式语音生成场景,兼顾速度与质量。

当前语音合成模型存在效率瓶颈:自回归(AR)模型生成慢,非自回归(NAR)模型因无序时序导致高延迟。为此,我们提出基于时序掩码变换器(TMT)的SyncSpeech模型,融合AR模型的时间有序性与NAR模型的并行解码效率。TMT通过精心设计的序列构建规则、训练目标和混合注意力掩码实现。为提升训练效率,引入高概率掩码策略,显著改善整体性能。推理时,同步处理每个新输入文本词对应的所有语音标记,单步完成;在接收第二个文本词后立即开始生成语音,实现低延迟。实验表明,SyncSpeech在保持与现代AR模型相当语音质量的同时,首包延迟降低5.8倍,实时因子提升8.8倍。语音样例见https://SyncSpeech.github.io/。

原文摘要 · Abstract (English)

Current text-to-speech (TTS) models face a persistent limitation: autoregressive (AR) models suffer from low generation efficiency, while modern non-autoregressive (NAR) models experience high latency due to their unordered temporal nature. To bridge this divide, we introduce SyncSpeech, an efficient and low-latency TTS model based on the proposed Temporal Mask Transformer (TMT) paradigm. TMT synergistically unifies the temporally ordered generation of AR models with the parallel decoding efficiency of NAR models. TMT is realized through a meticulously designed sequence construction rule, a corresponding training objective, and a specialized hybrid attention mask. Furthermore, with the primary aim of enhancing training efficiency, a high-probability masking strategy is introduced, which also leads to a significant improvement in overall model performance. During inference, SyncSpeech achieves high efficiency by decoding all speech tokens corresponding to each newly arrived text token in a single step, and low latency by beginning to generate speech immediately upon receiving the second text token from the streaming input. Evaluations show that SyncSpeech maintains speech quality comparable to the modern AR TTS model, while achieving a 5.8-fold reduction in first-packet latency and an 8.8-fold improvement in real-time factor. Speech samples are available at https://SyncSpeech.github.io/}{https://SyncSpeech.github.io/.

语音合成低延迟时序建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。