arXiv:2607.04140eess.AScs.CL2026-07

将自回归语音模型转为扩散式语言模型,提速3.3倍且减少生成错误。

DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech

论文配图:DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech
图 1 · 摘自论文原文
  • 用轻量LoRA框架将自回归模型转为扩散式解码,支持置信度排序生成。
  • 仅用585小时数据训练,语音识别错误率降至1.75%,生成速度提升3.3倍。
  • 改善文本-语音对齐,提高解码置信度,有效抑制自回归模型的幻觉问题。

自回归(AR)语音合成模型逐个生成离散语音标记,导致推理慢且鲁棒性差,局部错误会传播并放大成幻觉。这源于其从左到右的顺序依赖:每个标记必须在后续上下文出现前确定。但语音合成无需此限制,因模型在生成前已接收完整输入文本。本文提出DELTA-TTS,一种基于LoRA的轻量级适配框架,可将预训练的AR TTS模型转化为离散扩散语言模型(dLLM),实现置信度有序的语音标记解码。为更好捕捉语音局部结构,该方法引入卷积模块注入局部声学上下文,并采用1/t加权训练目标与时间偏移推理调度,将低置信度位置推迟到后期处理。在仅585小时的LibriTTS数据上训练后,DELTA-TTS在Seed-TTS测试集上达到1.75%的词错误率(WER),优于其自回归基线模型,同时生成速度提升3.3倍。进一步分析表明,该方法提升了文本-语音对齐精度,增强整体解码置信度,显著缓解了自回归生成中的幻觉现象。

原文摘要 · Abstract (English)

Autoregressive (AR) text-to-speech (TTS) models generate discrete speech tokens sequentially, which makes inference slow and can degrade robustness, since local errors propagate to later positions and can escalate into hallucination. This limitation stems from their left-to-right AR commitment: each token must be determined before future speech-token context is available. However, such ordering is not an inherent requirement for TTS, since the model receives the full input text before synthesis. In this paper, we introduce DELTA-TTS, a lightweight LoRA-based adaptation framework that converts a pretrained AR TTS model into a discrete diffusion language model (dLLM) for confidence-ordered speech-token decoding. To better capture the local structure of speech, DELTA-TTS incorporates a convolution module that injects local acoustic context, together with a 1/t-weighted training objective and a time-shifted inference schedule that together defer low-confidence positions to later steps. Trained on only 585 hours of LibriTTS, DELTA-TTS achieves a 1.75% WER on Seed-TTS test-en, outperforming its AR backbone while generating tokens 3.3x faster. Further analysis shows that DELTA-TTS produces sharper text--speech alignment, increases overall decoding confidence, and mitigates the hallucinations observed in AR generation.

语音合成扩散模型自回归LoRA

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。