让语音合成学会不确定性,生成更自然的语音。
Bayesian Speech Synthesizers Can Learn from Multiple Teachers

- 用贝叶斯方法建模语音生成的不确定性,不增加参数和延迟。
- 仅用5000小时数据超越5万小时训练的模型,WER降低25.8%。
- 适合追求高自然度语音生成与动态语调控制的研究者。
文本到语音(TTS)本质上是“一对多”的映射,具有内在不确定性,但当前范式常将其简化为确定性回归任务。尽管连续自回归(AR)模型作为离散编码器方法的替代方案逐渐兴起,但通常依赖固定方差先验,导致生成局限于静态点估计,忽略自然语音的动态变化。为此,我们提出BELLE(贝叶斯证据学习与语言建模),该框架从确定性预测转向严格的贝叶斯推断,且不增加模型参数或推理延迟。通过将声学目标建模为正态-逆伽马分布,BELLE捕捉数据相关的随机不确定性。为在标准单参考数据集上实现准确的方差估计,我们引入“一对多”训练策略,利用合成样本作为统计支持集,使模型学习稳健的分布特性,而非单纯模仿教师模型的噪声。实验表明,仅用约5000小时数据训练的BELLE,在相对词错误率(WER)上比领先开源模型(训练于5万小时数据)降低25.8%,并天然支持高质量流式生成。音频样本见 https://belletts.github.io/Belle/。
原文摘要 · Abstract (English)
Text-to-Speech (TTS) is inherently a "one-to-many" mapping characterized by intrinsic uncertainty, yet current paradigms often oversimplify it into a deterministic regression task. While continuous-valued autoregressive (AR) models have recently emerged as a promising alternative to discrete codec-based approaches, they typically rely on a fixed-variance prior, fundamentally constraining generation to a static point estimate that ignores the dynamic variability of natural speech. To bridge this gap, we propose BELLE (Bayesian evidential learning with language modelling), a framework that shifts from deterministic prediction to principled Bayesian inference without increasing model parameters or inference latency. By modeling the acoustic target as a Normal-Inverse-Gamma distribution, BELLE captures data-dependent aleatoric uncertainty. To enable accurate variance estimation on standard single-reference datasets, we introduce a "one-to-many" training strategy that leverages synthetic samples as a statistical support set, allowing the model to learn robust distributional properties rather than merely imitating teacher artifacts. Experiments demonstrate that BELLE, trained on only ~5k hours of data, outperforms leading open-source models trained on 50k hours (achieving a 25.8% relative WER reduction) and naturally supports high-quality streaming generation. Audio samples are available at https://belletts.github.io/Belle/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。