无需语音提示文本即可跨语言克隆声音，提升语音合成灵活性
Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis
- 用强制对齐提取音视频中的词边界，训练时无需参考文本
- 通过多粒度语速预测器生成说话节奏，实现无文本推理
- 支持未见语言的语音克隆，适合多语言语音应用开发
基于流匹配的文本到语音（TTS）模型已展现出高质量语音合成能力。然而，当前多数流匹配型TTS模型仍依赖与音频提示对应的参考文本进行合成，这在语音提示文本不可用时限制了跨语言语音克隆能力，尤其在未见语言场景下更为明显。主要挑战在于训练中如何识别词边界，以及推理中如何确定合适持续时间。本文提出Cross-Lingual F5-TTS框架，实现无需音频提示文本的跨语言语音克隆。方法上，通过强制对齐预处理音频提示以获取词边界，使模型可直接从音频提示合成，训练时排除文本输入；为解决持续时间建模难题，我们在不同语言粒度上训练语速预测器，从说话人语速推导出持续时间。实验表明，该方法性能与F5-TTS相当，同时支持跨语言语音克隆。
原文摘要 · Abstract (English)
Flow-matching-based text-to-speech (TTS) models have shown high-quality speech synthesis. However, most current flow-matching-based TTS models still rely on reference transcripts corresponding to the audio prompt for synthesis. This dependency prevents cross-lingual voice cloning when audio prompt transcripts are unavailable, particularly for unseen languages. The key challenges for flow-matching-based TTS models to remove audio prompt transcripts are identifying word boundaries during training and determining appropriate duration during inference. In this paper, we introduce Cross-Lingual F5-TTS, a framework that enables cross-lingual voice cloning without audio prompt transcripts. Our method preprocesses audio prompts by forced alignment to obtain word boundaries, enabling direct synthesis from audio prompts while excluding transcripts during training. To address the duration modeling challenge, we train speaking rate predictors at different linguistic granularities to derive duration from speaker pace. Experiments show that our approach matches the performance of F5-TTS while enabling cross-lingual voice cloning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。