用扩散模型实现低资源印地语语音合成，支持未知说话人零样本生成。
A2TTS: TTS for Low Resource Indian Languages
- 基于扩散模型的声码器，用短音频提取说话人嵌入来控制多说话人生成。
- 引入交叉注意力时长预测，提升语音韵律自然度与说话人一致性。
- 结合无分类器引导，显著改善未知说话人的语音合成质量。
我们提出一种面向未见说话人且支持多种印度语言的说话人条件化文本转语音（TTS）系统。该方法采用基于扩散的TTS架构，通过说话人编码器从短参考音频中提取嵌入，用于条件化DDPM解码器以实现多说话人语音生成。为进一步提升语音韵律与自然度，引入基于交叉注意力的时长预测机制，利用参考音频实现更准确、说话人一致的时长建模。该方法使生成语音更贴近目标说话人特征，同时改善时长建模与整体表现力。为增强零样本生成能力，采用无分类器引导策略，使系统在未知说话人情况下仍能生成更接近真实语音的输出。基于IndicSUPERB数据集，在包括孟加拉语、古吉拉特语、印地语、马拉地语、马拉雅拉姆语、旁遮普语和泰米尔语在内的多种印度语言上训练了语言特定的说话人条件化模型。
原文摘要 · Abstract (English)
We present a speaker conditioned text-to-speech (TTS) system aimed at addressing challenges in generating speech for unseen speakers and supporting diverse Indian languages. Our method leverages a diffusion-based TTS architecture, where a speaker encoder extracts embeddings from short reference audio samples to condition the DDPM decoder for multispeaker generation. To further enhance prosody and naturalness, we employ a cross-attention based duration prediction mechanism that utilizes reference audio, enabling more accurate and speaker consistent timing. This results in speech that closely resembles the target speaker while improving duration modeling and overall expressiveness. Additionally, to improve zero-shot generation, we employed classifier free guidance, allowing the system to generate speech more near speech for unknown speakers. Using this approach, we trained language-specific speaker-conditioned models. Using the IndicSUPERB dataset for multiple Indian languages such as Bengali, Gujarati, Hindi, Marathi, Malayalam, Punjabi and Tamil.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。