arXiv:2606.09234cs.SDcs.AI2026-06

端到端训练让语音生成模型更准更快,打破传统分步训练瓶颈。

End-to-End Training for Discrete Token LLM based TTS System

论文配图:End-to-End Training for Discrete Token LLM based TTS System
图 1 · 摘自论文原文
  • 统一训练语音分词器、大语言模型和流匹配模型,联合优化多任务目标。
  • 在Seed-TTS-Eval上达到0.78%词错误率,0.5B参数模型创新高。
  • 适合追求高效语音合成系统的研究者与开发者参考。

当前最先进的文本转语音(TTS)系统通常采用分阶段流水线:语音分词器、自回归大语言模型(LLM)和基于扩散的流匹配(FM)模型,各组件独立训练。本文提出一种全端到端(E2E)优化框架,统一训练语音分词器、LLM、FM模型及额外的奖励模型(RM)。首先,通过重建(用于FM)、下一词预测(用于LLM)和多识别任务(用于RM)的多任务目标联合优化分词器,使离散语音标记空间更好地捕捉声学与语义关键信息。随后,利用下游重建与识别任务进一步优化LLM,减少推理时偏差,引导生成更符合偏好结果。实验表明,本框架持续优于分阶段基线。在Seed-TTS-Eval基准上,使用0.6B参数LLM和0.5B参数FM模型,取得0.78%与1.56%的词错误率,为当前最先进水平。结果验证了整体端到端优化对离散标记语音合成系统的重要性,且训练流程大幅简化。

原文摘要 · Abstract (English)

Recent state-of-the-art (SOTA) text-to-speech (TTS) systems typically adopt a cascaded pipeline consisting of a speech tokenizer, an autoregressive large language model (LLM), and a diffusion based flow-matching (FM) model, with these components trained independently. In this paper, we propose a fully end-to-end (E2E) optimization framework that unifies the training of the speech tokenizer, LLM, FM model, and an additional reward model (RM). Specifically, we first jointly optimize the tokenizer using multi-task objectives derived from reconstruction for FM, next-token prediction for LLM, and multi recognition task for RM. This joint training encourages the discrete speech token space to capture acoustically and semantically salient information that is better tailored to TTS. We then further optimize the LLM using downstream reconstruction and recognition by FM and RM, which reduces inference-time mismatch and steers the LLM toward more preferred generations. Experimental results show that our E2E framework consistently outperforms cascaded baselines. On the Seed-TTS-Eval benchmark, our system achieves a word error rate (WER) of 0.78% and 1.56%, a new SOTA result with a 0.6B-parameter LLM and 0.5B-parameter FM model. These results validate that holistic E2E optimization is critical for improving discrete-token-based TTS systems with a much simpler training pipeline.

语音合成端到端大语言模型流匹配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。