FlexSpeech实现稳定且可控制的语音合成,支持快速风格迁移。
FlexSpeech: Towards Stable, Controllable and Expressive Text-to-Speech
- 分离时长预测与声学建模,结合自回归与非自回归优势。
- 零样本语音合成达最新水平,仅用100样本即可完成风格迁移。
- 适合需要快速适配新语调或风格的语音应用开发。
当前语音生成研究主要分为非自回归(NAR)与自回归(AR)两类,核心差异在于时长预测策略。NAR方法通过显式独立建模每个音素时长确保生成稳定性;而AR方法则利用马尔可夫特性隐式建模时长,虽提升韵律自然度,但缺乏结构稳定性保障。为兼顾稳定性与自然性,本文提出FlexSpeech,一种稳定、可控且富有表现力的文本到语音模型。其核心思路是将语音生成拆分为自回归时长预测器与非自回归声学模型两部分:声学模型在大量数据上训练,以稳定生成受参考语音韵律和音素时长引导的音频;时长预测器则通过轻量级优化适配不同风格,实现与说话人音色解耦的快速风格迁移。实验表明,该方法在零样本语音合成中达到当前最优的稳定性和自然度。更重要的是,在特定风格领域迁移时,仅需约100个样本即可完成时长模块的轻量化优化,无需调整声学模型,实现高效稳定的风格转换。
原文摘要 · Abstract (English)
Current speech generation research can be categorized into two primary classes: non-autoregressive and autoregressive. The fundamental distinction between these approaches lies in the duration prediction strategy employed for predictable-length sequences. The NAR methods ensure stability in speech generation by explicitly and independently modeling the duration of each phonetic unit. Conversely, AR methods employ an autoregressive paradigm to predict the compressed speech token by implicitly modeling duration with Markov properties. Although this approach improves prosody, it does not provide the structural guarantees necessary for stability. To simultaneously address the issues of stability and naturalness in speech generation, we propose FlexSpeech, a stable, controllable, and expressive TTS model. The motivation behind FlexSpeech is to incorporate Markov dependencies and preference optimization directly on the duration predictor to boost its naturalness while maintaining explicit modeling of the phonetic units to ensure stability. Specifically, we decompose the speech generation task into two components: an AR duration predictor and a NAR acoustic model. The acoustic model is trained on a substantial amount of data to learn to render audio more stably, given reference audio prosody and phone durations. The duration predictor is optimized in a lightweight manner for different stylistic variations, thereby enabling rapid style transfer while maintaining a decoupled relationship with the specified speaker timbre. Experimental results demonstrate that our approach achieves SOTA stability and naturalness in zero-shot TTS. More importantly, when transferring to a specific stylistic domain, we can accomplish lightweight optimization of the duration module solely with about 100 data samples, without the need to adjust the acoustic model, thereby enabling rapid and stable style transfer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。