用可微分奖励优化提升语音合成准确率，支持零样本情感控制。
Differentiable Reward Optimization for LLM based TTS system
- 基于编码器令牌直接计算可微分奖励，替代传统音频反馈。
- 在seed-tts-eval上达到当前最优的语音识别错误率（WER）。
- 多任务奖励模型实现零样本情绪与质量控制，适合语音系统优化场景。
本文提出一种新型可微分奖励优化（DiffRO）方法，用于提升基于神经编解码语言模型的文本到语音（TTS）系统性能。与传统基于人类反馈强化学习（RLHF）不同，DiffRO直接基于神经编解码令牌计算奖励，而非依赖合成音频。同时，采用Gumbel-Softmax技术使奖励函数可微，简化了RLHF训练流程。此外，引入多任务奖励（MTR）模型，从多个角度提供反馈，显著增强系统对指令的遵循能力。实验表明，DiffRO显著提升TTS系统的发音准确性，在seed-tts-eval基准上达到当前最优（SOTA）的词错误率（WER）。结合MTR模型后，系统可在零样本条件下有效控制情感与音质属性。
原文摘要 · Abstract (English)
This paper proposes a novel Differentiable Reward Optimization (DiffRO) method aimed at enhancing the performance of neural codec language models based text-to-speech (TTS) systems. In contrast to conventional reinforcement learning from human feedback (RLHF) approaches applied to TTS, DiffRO directly compute the rewards based on neural codec tokens, rather than relying on synthesized audio. Furthermore, we employ the Gumbel-Softmax technique to render the reward function differentiable, thereby streamlining the RLHF training process. Additionally, we introduce a multi-task reward (MTR) model which can provide feedback from different perspectives and find that it can augment the system's capability to follow instructions effectively.Experimental results indicate that DiffRO significantly improves the pronunciation accuracy of the TTS system, achieving state-of-the-art (SOTA) WER results on the seed-tts-eval benchmark. Moreover, with the integration of the MTR model, we demonstrate the ability to control emotional and quality attributes in a zero-shot manner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。