用扩散模型替代自回归生成，实现语音合成加速与零样本编辑。
LLaDA-TTS: Unifying Speech Synthesis and Zero-Shot Editing via Masked Diffusion Modeling
- 用掩码扩散模型并行生成语音，固定64步完成，不依赖序列长度。
- 仅用50小时数据微调，中文词错误率0.98%，英文词错误率1.96%，速度提升2倍。
- 无需额外训练即可实现词级插入、删除、替换，适合快速原型开发。
基于大语言模型的语音合成系统通过自回归解码实现了出色自然度，但需依次生成N个语音标记，耗时与序列长度成正比。我们提出LLaDA-TTS，将自回归语言模型替换为掩码扩散模型，可在固定64步内并行完成生成，使推理延迟与序列长度解耦。令人惊讶的是，仅用50小时微调数据，我们通过双向注意力成功将预训练的自回归检查点迁移到掩码扩散范式。在Seed-TTS-Eval上，64步下中文词错误率（CER）达0.98%，英文词错误率（WER）为1.96%，性能匹配原始CosyVoice 3基线，同时实现2倍于原自回归模型的生成速度——尽管未使用键值缓存（KV cache），而该优化是自回归基线的关键加速手段。此外，双向架构天然支持零样本语音编辑，包括词级插入、删除和替换，无需额外训练。理论上，我们证明在声学标记的局部性假设下，自回归预训练权重对双向掩码预测接近最优，解释了快速收敛现象。该方法仅修改注意力掩码和目标函数，可无缝适配任意基于语言模型的自回归语音合成系统。代码与音频样本将公开于https://deft-piroshki-b652b6.netlify.app/。
原文摘要 · Abstract (English)
Large language model (LLM)-based text-to-speech (TTS) systems achieve remarkable naturalness via autoregressive (AR) decoding, but require N sequential steps to generate N speech tokens. We present LLaDA-TTS, which replaces the AR LLM with a masked diffusion model that completes generation in a fixed number of parallel steps, decoupling inference latency from sequence length. Remarkably, using only 50 hours of fine-tuning data, we successfully transfer a pretrained AR checkpoint to the masked diffusion paradigm via bidirectional attention. At 64 steps, LLaDA-TTS achieves 0.98% CER (zh) and 1.96% WER (en) on Seed-TTS-Eval, matching the original CosyVoice 3 baseline performance while delivering a 2x LLM-stage speedup--a notable acceleration achieved despite the absence of KV cache, an optimization the AR baseline heavily relies on. Beyond acceleration, the bidirectional architecture naturally enables zero-shot speech editing--including word-level insertion, deletion, and substitution--without any additional training. Theoretically, we prove that AR-pretrained weights are near-optimal for bidirectional masked prediction under the locality property of acoustic tokens, explaining this rapid convergence. This general method modifies only the attention mask and objective, applying seamlessly to any LLM-based AR TTS system. Code and audio samples will be available at https://deft-piroshki-b652b5.netlify.app/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。