KALL-E用连续语音分布直接生成语音，无需扩散模型。
KALL-E:Autoregressive Speech Synthesis with Next-Distribution Prediction
- 用Flow-VAE提取连续语音表征，再由自回归Transformer预测分布。
- 在LJSpeech上实现自然度9.1、清晰度8.9，单样本即可适配新音色。
- 适合追求高保真语音合成与快速音色迁移的研究者。
我们提出KALL-E，一种新型自回归语言模型用于文本到语音合成，通过预测连续语音帧的下一分布来工作。与现有方法不同，KALL-E直接建模条件于文本的连续语音分布，无需任何基于扩散的组件。具体而言，我们使用Flow-VAE从波形中提取连续语音表征，而非依赖离散语音标记。一个单一的自回归Transformer被训练以从文本预测这些连续语音分布，并以KL散度损失为优化目标。实验结果表明，KALL-E实现了更优的语音合成质量，甚至仅需单个样本即可适应目标说话人。重要的是，KALL-E为在语音合成中利用连续语音表征提供了更直接、更有效的方法。
原文摘要 · Abstract (English)
We introduce KALL-E, a novel autoregressive (AR) language model for text-to-speech (TTS) synthesis that operates by predicting the next distribution of continuous speech frames. Unlike existing methods, KALL-E directly models the continuous speech distribution conditioned on text, eliminating the need for any diffusion-based components. Specifically, we utilize a Flow-VAE to extract a continuous latent speech representation from waveforms, instead of relying on discrete speech tokens. A single AR Transformer is then trained to predict these continuous speech distributions from text, optimizing a Kullback-Leibler divergence loss as its objective. Experimental results demonstrate that KALL-E achieves superior speech synthesis quality and can even adapt to a target speaker from just a single sample. Importantly, KALL-E provides a more direct and effective approach for utilizing continuous speech representations in TTS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。