无需语音样本,即可跨语言迁移个人声音,还能恢复失语者语音。
Zero-shot Cross-lingual Voice Transfer for TTS
- 用参考语音和残差适配器实现零样本跨语言声线迁移
- 单个英语语音样本使跨9种语言声线相似度达73%
- 特别适合无法提供正常语音的失语者语音重建
本文提出一种零样本语音迁移(VT)模块,可无缝集成至多语言文本转语音(TTS)系统中,实现个人声音在不同语言间的迁移。该模块包含处理参考语音的说话人编码器、瓶颈层与残差适配器,连接现有TTS层。我们对比了多种组件配置,报告了平均意见分(MOS)和跨语言说话人相似性。仅需每名说话人一个英语参考语音,即可在九种目标语言上实现平均73%的语音迁移相似度。语音特征对个体身份的构建与感知至关重要。因生理或神经疾病导致失声,会严重冲击个体核心身份认同。作为案例研究,我们证明该方法不仅能迁移正常语音,还可基于异常语音样本恢复失语者语音——对从未拥有正常语音或未提前录制语音的人群具有重要价值。典型跨语言语音样本及失语者语音恢复演示视频详见:google.github.io/tacotron/publications/zero_shot_voice_transfer。
原文摘要 · Abstract (English)
In this paper, we introduce a zero-shot Voice Transfer (VT) module that can be seamlessly integrated into a multi-lingual Text-to-speech (TTS) system to transfer an individual's voice across languages. Our proposed VT module comprises a speaker-encoder that processes reference speech, a bottleneck layer, and residual adapters, connected to preexisting TTS layers. We compare the performance of various configurations of these components and report Mean Opinion Score (MOS) and Speaker Similarity across languages. Using a single English reference speech per speaker, we achieve an average voice transfer similarity score of 73% across nine target languages. Vocal characteristics contribute significantly to the construction and perception of individual identity. The loss of one's voice, due to physical or neurological conditions, can lead to a profound sense of loss, impacting one's core identity. As a case study, we demonstrate that our approach can not only transfer typical speech but also restore the voices of individuals with dysarthria, even when only atypical speech samples are available - a valuable utility for those who have never had typical speech or banked their voice. Cross-lingual typical audio samples, plus videos demonstrating voice restoration for dysarthric speakers are available here (google.github.io/tacotron/publications/zero_shot_voice_transfer).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。