ZipVoice用流匹配实现快速高质零样本语音合成。
ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching
- 基于压缩版模型架构，保持小体积下强建模能力。
- 比同类模型小3倍、快30倍，语音质量相当。
- 适合追求高效部署的语音合成应用开发。
现有大规模零样本语音合成模型虽具备高语音质量，但因参数量过大导致推理速度慢。为此，本文提出ZipVoice，一种基于流匹配的零样本语音合成模型，具有紧凑模型规模和快速推理速度。核心设计包括：1）采用基于Zipformer的向量场估计器，在模型尺寸受限时仍保持足够的建模能力；2）使用平均上采样实现初始语音-文本对齐，结合Zipformer文本编码器提升语音可懂度；3）引入流蒸馏方法减少采样步数，并消除无分类器引导带来的推理开销。在10万小时多语言数据集上的实验表明，ZipVoice在语音质量上达到当前最优水平，同时模型体积比基于DiT的流匹配基线小3倍，推理速度最快可达其30倍。代码、模型检查点及演示样本已公开于https://github.com/k2-fsa/ZipVoice。
原文摘要 · Abstract (English)
Existing large-scale zero-shot text-to-speech (TTS) models deliver high speech quality but suffer from slow inference speeds due to massive parameters. To address this issue, this paper introduces ZipVoice, a high-quality flow-matching-based zero-shot TTS model with a compact model size and fast inference speed. Key designs include: 1) a Zipformer-based vector field estimator to maintain adequate modeling capabilities under constrained size; 2) Average upsampling-based initial speech-text alignment and Zipformer-based text encoder to improve speech intelligibility; 3) A flow distillation method to reduce sampling steps and eliminate the inference overhead associated with classifier-free guidance. Experiments on 100k hours multilingual datasets show that ZipVoice matches state-of-the-art models in speech quality, while being 3 times smaller and up to 30 times faster than a DiT-based flow-matching baseline. Codes, model checkpoints and demo samples are publicly available at https://github.com/k2-fsa/ZipVoice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。