通过改进初始分布,让语音克隆模型生成更快更准。
SF-Speech: Straightened Flow for Zero-Shot Voice Clone
- 用多阶段模块生成更确定的初始分布
- 仅用1/4步数实现超快生成,速度是Voicebox的3.7倍
- 无需额外损失函数,适合快速语音克隆应用
近期基于流匹配训练的神经常微分方程(ODE)模型在零样本语音克隆任务中表现优异。然而,假设标准高斯噪声为ODE的初始分布会导致流匹配目标存在大量交叉,增加模型训练难度并提升生成轨迹的曲率,限制了少步生成能力。本文提出SF-Speech,一种基于ODE和上下文学习的新型语音克隆模型。不同于以往方法,SF-Speech采用轻量级多阶段模块生成更确定的初始分布,在不引入额外损失函数的前提下,通过联合训练有效拉直了ODE模型的逆向轨迹。在多种规模数据集上的实验表明,SF-Speech超越现有最先进零样本TTS方法,仅需四分之一的求解器步数,生成速度约为Voicebox和E2 TTS的3.7倍。音频样本可访问演示页。
原文摘要 · Abstract (English)
Recently, neural ordinary differential equations (ODE) models trained with flow matching have achieved impressive performance on the zero-shot voice clone task. Nevertheless, postulating standard Gaussian noise as the initial distribution of ODE gives rise to numerous intersections within the fitted targets of flow matching, which presents challenges to model training and enhances the curvature of the learned generated trajectories. These curved trajectories restrict the capacity of ODE models for generating desirable samples with a few steps. This paper proposes SF-Speech, a novel voice clone model based on ODE and in-context learning. Unlike the previous works, SF-Speech adopts a lightweight multi-stage module to generate a more deterministic initial distribution for ODE. Without introducing any additional loss function, we effectively straighten the curved reverse trajectories of the ODE model by jointly training it with the proposed module. Experiment results on datasets of various scales show that SF-Speech outperforms the state-of-the-art zero-shot TTS methods and requires only a quarter of the solver steps, resulting in a generation speed approximately 3.7 times that of Voicebox and E2 TTS. Audio samples are available at the demo page\footnote{[Online] Available: https://lixuyuan102.github.io/Demo/}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。