arXiv:2608.03215eess.AScs.AI2026-08

用分组优势加权强化学习优化语音生成,提升自然度与相似度。

GROW: Group-Relative Advantage-Weighted On-Policy Reinforcement Learning of Autoregressive-Diffusion Text-to-Speech model

  • 基于分组内相对优势加权,直接优化流匹配目标
  • 在LibriSpeech上将平均WER降至1.558,说话人相似度升至0.715
  • 训练速度比32-NFE基线快2.9倍,支持开源复现

基于流匹配的文本到语音生成面临确定性微分方程采样带来的强化学习复杂性:传统轨迹级策略梯度方法通常将微分方程转为随机微分方程并追踪每步似然比,引入随机扰动和高开销。本文提出GROW,一种针对标准流匹配目标的分组相对优势加权在线强化学习方法。对于每个提示,GROW采样一组自洽语音,分别对组内可懂度与说话人相似度奖励进行标准化,并合并用于重加权流匹配回归。通过Wasserstein-2速度正则项锚定更新模型至冻结预训练参考模型。引入组均值奖励基准,将奖励加权转化为优势加权。对于奖励集中度高的强预训练模型,正指数加权易被无差别的自我模仿主导,而零均值有符号优势能实现有效的组内信用分配。在DiTAR上实现,在LibriSpeech和Seed-TTS EN/ZH上评估,平均字错误率从2.016降至1.558,说话人相似度从0.676升至0.715,同时保持UTMOS不变。采用10-NFE训练滚动步长与32-NFE推理,性能相当,但训练速度比32-NFE DiTAR-GRPO快2.9倍。项目将开源完整GROW代码、忠实的DiTAR复现及所有模型检查点。

原文摘要 · Abstract (English)

Reinforcement learning for flow-matching text-to-speech is complicated by deterministic ODE sampling: trajectory-level policy-gradient methods typically convert the ODE into an SDE and track per-step likelihood ratios, introducing stochastic perturbations and substantial overhead. We propose GROW, a group-relative advantage-weighted on-policy RL method that acts directly on the standard flow-matching objective. For each prompt, GROW samples a group of on-policy utterances, separately standardizes intelligibility and speaker-similarity rewards within the group, and combines them to reweight flow-matching regression. A Wasserstein-2 velocity penalty anchors the updated model to a frozen pretrained reference. A group-mean reward baseline is introduced to convert reward weighting into advantage weighting. For strong pretrained TTS models with concentrated rewards, positive exponential weighting is dominated by reward-agnostic self-imitation, whereas a zero-mean signed advantage preserves effective within-group credit assignment. Instantiated on DiTAR and evaluated on LibriSpeech and Seed-TTS EN/ZH, GROW reduces average WER from 2.016 to 1.558 and raises speaker similarity from 0.676 to 0.715 while keeping UTMOS. With 10-NFE training rollouts and 32-NFE evaluation, GROW retains comparable performance while training 2.9x faster than 32-NFE DiTAR-GRPO. We will open-source complete GROW codes, faithful DiTAR reproduction, and all model checkpoints.

语音合成强化学习流匹配高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。