无需训练数据，一键实现跨语言任意语音转换
EZ-VC: Easy Zero-shot Any-to-Any Voice Conversion
- 用自监督模型提取离散语音表示，搭配非自回归扩散变换器解码
- 纯无文本自监督训练，零样本跨语言转换效果显著提升
- 适合语音克隆、多语种内容生成等场景，无需额外数据
近年来语音转换研究日益关注零样本能力的提升。尽管取得显著进展，现有方法在零样本跨语言场景中仍表现不佳，难以泛化到未见语言和口音的说话人。本文提出一种简单而有效的方法：结合自监督模型的离散语音表示与基于条件流匹配的非自回归扩散变换器解码器。该架构可实现完全无文本、自监督的语音转换模型训练，无需多个编码器分离语音特征。模型在零样本跨语言设置下，即便面对未见语言也能表现优异。演示地址：https://ez-vc.github.io/EZ-VC-Demo/
原文摘要 · Abstract (English)
Voice Conversion research in recent times has increasingly focused on improving the zero-shot capabilities of existing methods. Despite remarkable advancements, current architectures still tend to struggle in zero-shot cross-lingual settings. They are also often unable to generalize for speakers of unseen languages and accents. In this paper, we adopt a simple yet effective approach that combines discrete speech representations from self-supervised models with a non-autoregressive Diffusion-Transformer based conditional flow matching speech decoder. We show that this architecture allows us to train a voice-conversion model in a purely textless, self-supervised fashion. Our technique works without requiring multiple encoders to disentangle speech features. Our model also manages to excel in zero-shot cross-lingual settings even for unseen languages. For Demo: https://ez-vc.github.io/EZ-VC-Demo/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。