arXiv:2410.06885eess.AScs.SD2024-10ACL被引 550

F5-TTS用流匹配实现快速自然的多语言语音合成,无需复杂对齐设计。

F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

  • 用ConvNeXt增强文本表征并结合流匹配直接生成语音
  • 推理时采用Sway采样策略,推理实时率达0.15,速度显著提升
  • 支持零样本生成、无缝语码转换,适合语音合成研究者使用

本文提出F5-TTS,一种基于流匹配与扩散Transformer(DiT)的全非自回归语音合成系统。无需持续时间模型、文本编码器或音素对齐,仅将文本输入通过填充标记补长至与语音相同长度,再进行去噪生成,该思路在E2 TTS中已被验证可行。但原E2 TTS存在收敛慢、鲁棒性差的问题。为此,我们首先引入ConvNeXt对输入文本建模,增强其与语音的对齐能力;进一步提出推理阶段的Sway采样策略,显著提升模型性能与效率,且可无须重训练应用于其他流匹配模型。该设计实现更快训练,推理实时率(RTF)达0.15,远超现有基于扩散的语音合成模型。在10万小时多语言公开数据集上训练后,F5-TTS展现出高度自然、富有表现力的零样本生成能力、无缝语码转换及速度控制效率。代码与模型权重已开源:https://SWivid.github.io/F5-TTS/

原文摘要 · Abstract (English)

This paper introduces F5-TTS, a fully non-autoregressive text-to-speech system based on flow matching with Diffusion Transformer (DiT). Without requiring complex designs such as duration model, text encoder, and phoneme alignment, the text input is simply padded with filler tokens to the same length as input speech, and then the denoising is performed for speech generation, which was originally proved feasible by E2 TTS. However, the original design of E2 TTS makes it hard to follow due to its slow convergence and low robustness. To address these issues, we first model the input with ConvNeXt to refine the text representation, making it easy to align with the speech. We further propose an inference-time Sway Sampling strategy, which significantly improves our model's performance and efficiency. This sampling strategy for flow step can be easily applied to existing flow matching based models without retraining. Our design allows faster training and achieves an inference RTF of 0.15, which is greatly improved compared to state-of-the-art diffusion-based TTS models. Trained on a public 100K hours multilingual dataset, our F5-TTS exhibits highly natural and expressive zero-shot ability, seamless code-switching capability, and speed control efficiency. We have released all codes and checkpoints to promote community development, at https://SWivid.github.io/F5-TTS/.

语音合成流匹配非自回归多语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。