用强化学习提升语音模型表现，小数据也能见效。
Explore the Reinforcement Learning for the LLM based ASR and TTS system
- 设计轻量级强化学习框架，支持音频输入输出处理。
- 在少量数据和少步优化下，语音识别与合成性能显著提升。
- 适合研究语音生成与智能语音系统优化的开发者。
近年来，大语言模型（LLMs）在自动语音识别（ASR）和文本转语音（TTS）系统中发挥重要作用。尽管强化学习（RL）已在文本任务中显著提升LLM性能，但其在语音模型中的应用仍因训练复杂性而未被充分探索。本文提出一种专为音频型LLM设计的轻量级强化学习框架，可处理音频输入并生成音频输出。基于该框架，我们评估了强化学习在ASR与TTS任务中的有效性。在ASR任务中，我们在组相对策略优化（GRPO）框架下测试多种规则奖励函数，并研究了强化学习数据构建的影响。在TTS任务中，我们对比了GRPO与可微分奖励优化（DiffRO），并进一步结合两者以获得更优性能。实验表明，即使在有限训练数据和较少优化步骤条件下，强化学习仍能显著提升两类系统的性能。
原文摘要 · Abstract (English)
In recent years, large language models (LLMs) have played an important role in automatic speech recognition (ASR) and text-to-speech (TTS) systems. While reinforcement learning (RL) has significantly enhanced LLM performance in text-based tasks, its application to ASR and TTS remains underexplored due to the complexity of training audio-based models. In this study, we propose a lightweight RL framework tailored for audio-based LLMs that can process audio inputs and generate audio outputs. Based on this framework, we evaluate the effectiveness of reinforcement learning on both ASR and TTS tasks. For the ASR task, we experiment with different rule-based reward functions within the Group Relative Policy Optimization (GRPO) framework and investigate the impact of RL data construction. For the TTS task, we compare GRPO with Differentiable Reward Optimization (DiffRO) and further combine the two approaches to achieve improved performance. Our experiments demonstrate that RL can significantly enhance the performance of both ASR and TTS systems, even with limited training data and a small number of optimization steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。