FireRedTTS-1S实现低延迟高保真流式语音合成,支持零样本克隆。
FireRedTTS-1S: An Upgraded Streamable Foundation Text-to-Speech System
- 分两步生成:文本转语义标记,再逐块/逐帧转语音。
- 最低0.1实时因子(RTF),300ms延迟;或0.3 RTF,150ms延迟。
- 可零样本克隆,音质媲美真实录音,适合实时语音应用。
本文将FireRedTTS升级为新版本FireRedTTS-1S,一种高质量流式基础文本到语音系统。该系统通过两阶段实现流式语音生成:文本到语义解码与语义到声学解码。在文本到语义解码中,语义感知语音分词器将语音信号转换为语义标记,可通过语言模型自回归生成。同时,语义到声学解码模块以流式方式将生成的语义标记转化为语音信号。我们实现了两种方法:1)基于分块的流式匹配流方法,2)多流语言模型方法。两者均实现高质量、可流式语音生成,但在实时因子(RTF)和延迟上不同:流匹配解码以块为单位生成,RTF低至0.1,延迟300ms;多流语言模型以帧为单位自回归生成,RTF为0.3,延迟仅150ms。在零样本语音克隆实验中,客观结果验证FireRedTTS-1S作为高质量基础模型,其可懂性与说话人相似性媲美工业基线系统。主观评分显示其合成效果接近真实录音,验证了其作为高质量流式基础TTS系统的有效性。
原文摘要 · Abstract (English)
In this work, we upgrade FireRedTTS to a new version, FireRedTTS-1S, a high-quality streaming foundation text-to-speech system. FireRedTTS-1S achieves streaming speech generation via two steps: text-to-semantic decoding and semantic-to-acoustic decoding. In text-to-semantic decoding, a semantic-aware speech tokenizer converts the speech signal into semantic tokens, which can be synthesized from the text via a language model in an auto-regressive manner. Meanwhile, the semantic-to-acoustic decoding module simultaneously translates generated semantic tokens into the speech signal in a streaming way. We implement two approaches to achieve this module: 1) a chunk-wise streamable flow-matching approach, and 2) a multi-stream language model-based approach. They both present high-quality and streamable speech generation but differ in real-time factor (RTF) and latency. Specifically, flow-matching decoding can generate speech by chunks, presenting a lower RTF of 0.1 but a higher latency of 300ms. Instead, the multi-stream language model generates speech by frames in an autoregressive manner, presenting a higher RTF of 0.3 but a low latency of 150ms. In experiments on zero-shot voice cloning, the objective results validate FireRedTTS-1S as a high-quality foundation model with comparable intelligibility and speaker similarity over industrial baseline systems. Furthermore, the subjective score of FireRedTTS-1S highlights its impressive synthesis performance, achieving comparable quality to the ground-truth recordings. These results validate FireRedTTS-1S as a high-quality streaming foundation TTS system.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。