基于扩散模型的多语言语音合成系统,支持零样本变声与编辑。
Luna-TTS Family Technical Report
- 采用扩散语言模型,非自回归并行生成语音编码网格。
- 端到端实时率0.024,首块延迟仅41.6毫秒,支持增量输出。
- 可精准控制情绪与非语言发声,性能超越主流开源与商用系统。
现代文本转语音(TTS)主要依赖自回归(AR)编解码语言模型,其从左到右的解码方式导致延迟随语句长度增加、前缀错误累积,并对残差向量量化(RVQ)标记网格施加人为生成顺序。本文提出Luna-TTS家族,基于扩散语言模型,利用涵盖中文、英文、日文和韩文的100万小时语音数据预训练。该系列通过逐步适应预训练的自回归文本语言模型,从因果注意力过渡到双向,最终采用块因果注意力,包含两个共享单个分词器、数据流水线及0.6B骨干模型的变体。Luna-TTS为全非自回归:在固定数量的并行精炼步骤中生成整个RVQ标记网格,零样本变声与语音编辑作为填充任务天然实现。Luna-TTS Realtime通过持续训练得到,在每32个编解码帧(1.28秒)上保持自回归,同时并行去噪;支持键值缓存的块状生成与增量音频传输,温启动协议下端到端实时率(RTF)达0.0240,首块本地延迟为41.6毫秒。经过退火微调阶段,显式控制情绪与非语言发声(NVVs),强化学习阶段采用GRPO,策略比由实际去噪轨迹计算得出。在Seed-TTS-Eval上,Luna-TTS在所有四项指标上均优于对比的开源与商用系统(测试中文:0.73 CER / 79.7 SIM;测试英文:1.49 WER / 76.8 SIM);在更具挑战性的in-the-wild CV3-Eval中,其普通话与英文错误率最低。相比领先商用系统,它在大多数客观、模型评估及人工评分指标上对NVV和情绪控制表现最优。
原文摘要 · Abstract (English)
Modern text-to-speech (TTS) is dominated by autoregressive (AR) codec language models, whose left-to-right decoding brings latency that grows with utterance length, error accumulation along the committed prefix, and an artificial generation order imposed on the Residual Vector Quantization (RVQ) token grid. We propose Luna-TTS Family, diffusion-language-model-based TTS systems pretrained on 1 million hours of speech across Chinese, English, Japanese, and Korean. The family is built by progressive adaptation of a pretrained AR text LLM, from causal to bidirectional and finally to block-causal attention, and comprises two variants sharing a single tokenizer, data pipeline, and 0.6B backbone lineage. Luna-TTS is fully non-autoregressive: it generates the entire RVQ token grid in a fixed number of parallel refinement steps, with zero-shot voice cloning and speech editing arising natively as infilling. Luna-TTS Realtime, derived by continual training, is autoregressive over blocks of 32 codec frames (1.28s) while denoising each block in parallel; it supports KV-cached blockwise generation and incremental audio delivery, achieving an end-to-end RTF of 0.0240 and 41.6 ms local first-block latency under the warmed serving protocol. An annealed fine-tuning stage adds explicit control over emotion and non-verbal vocalizations (NVVs), and a reinforcement-learning stage applies GRPO with policy ratios computed over the realized denoising trajectory. On Seed-TTS-Eval, Luna-TTS achieves the best results on all four metrics among compared open-source and commercial systems (0.73 CER / 79.7 SIM on test-zh, 1.49 WER / 76.8 SIM on test-en); on the harder in-the-wild CV3-Eval, it posts the lowest Mandarin and English error rates in our comparison. Against leading commercial systems, it achieves the best results on most objective, model-based, and human-rated metrics for NVV and emotion control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。