提出新方法提升无评判器强化学习的稳定性,解决奖励一致时无法学习的问题。
BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards

- 用历史统计与当前局部统计加权融合,稳定优势估计
- 在二元验证场景下训练更稳定,避免零方差导致学习停滞
- 适合冷启动阶段或奖励信号不充分的对齐任务
无评判器强化学习结合可验证奖励(RLVR)如组相对策略优化(GRPO),无需训练价值函数,降低大语言模型对齐的内存与计算开销。但GRPO式的优势估计依赖提示内奖励统计,当同一提示组所有回溯奖励相同时,组内方差为零,归一化后优势为零,导致冷启动阶段学习停滞。本文提出BV-Blend,通过结合提示内策略统计与语义聚类的历史均值和方差,利用均值标准误(SEM)作为置信度权重,动态融合历史与当前统计量,生成标准化优势用于PPO式裁剪更新。在可验证推理基准上的实验表明,该方法显著提升训练稳定性与性能,在组归一化方法可能停滞的场景中依然稳健。
原文摘要 · Abstract (English)
Critic-free reinforcement learning with verifiable rewards (RLVR), exemplified by Group Relative Policy Optimization (GRPO), avoids training a value function (critic) and reduces memory and compute overhead relative to critic-based PPO pipelines for aligning large language models. However, GRPO-style advantage estimation depends on prompt-local (within-prompt-group) reward statistics and can be unstable. In particular, when all rollouts in a prompt group receive identical rewards, the within-group reward variance becomes zero, and group normalization yields zero advantages for that group, impeding learning in cold-start regimes with binary verifiers. We introduce BV-Blend, a critic-free framework that stabilizes advantage estimation by combining prompt-local on-policy statistics with semantic-cluster-conditioned historical moments. BV-Blend maintains EMA-tracked reward moments for each cluster, derives a confidence weight from a standard error of the mean (SEM) proxy, and uses this weight to blend historical and prompt-local baseline and variance statistics into a standardized advantage for PPO-style clipped updates. Experiments on verifiable reasoning benchmarks show that BV-Blend improves training stability and performance, and remains robust in regimes where group-normalized methods may stall.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。