arXiv:2511.03710cs.LG2025-11被引 14

用收缩估计降低强化学习梯度方差,提升训练稳定性。

Shrinking the Variance: Shrinkage Baselines for Reinforcement Learning with Verifiable Rewards

  • 基于斯坦因悖论设计收缩估计器,融合提示内与跨提示均值
  • 在低生成样本下显著降低梯度方差,提升训练稳定性
  • 无需额外超参数,可直接替换现有基线,适合大模型后训练

基于可验证奖励的强化学习(RLVR)已成为使用策略梯度方法(如GRPO)对大型推理模型进行后训练的强大范式。为稳定训练,这些方法通常通过减去每个提示的采样均值来中心化轨迹奖励。统计上,这种中心化相当于控制变量(基线),可降低策略梯度估计器的方差。实践中,该均值通过批处理中每提示生成结果的采样平均估计。受斯坦因悖论启发,我们提出收缩估计器,将每提示均值与跨提示均值结合,以提高低生成样本情形下的每提示均值估计精度。理论上,我们构建了一个基于收缩的基线,可证明在各类算法中均产生更低方差的策略梯度估计。该基线是标准每提示均值基线的即插即用替代品,无需额外超参数或计算开销。实验表明,收缩基线持续优于经验均值基线,实现更低方差的梯度更新和更优的训练稳定性。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful paradigm for post-training large reasoning models (LRMs) using policy-gradient methods such as GRPO. To stabilize training, these methods typically center trajectory rewards by subtracting the empirical mean reward for each prompt. Statistically, this centering acts as a control variate (baseline), reducing the variance of the policy-gradient estimator. In practice, the mean reward is estimated using per-prompt empirical averages computed from the generations for each prompt in a batch. Motivated by Stein's paradox, we propose shrinkage estimators that combine per-prompt and across-prompt means to improve per-prompt mean estimation accuracy, especially in the low-generation regime typical of RLVR. Theoretically, we construct a shrinkage-based baseline that provably yields lower-variance policy-gradient estimators across algorithms. Our baseline is a drop-in replacement for standard per-prompt mean baselines and requires no additional hyperparameters or computation. Empirically, shrinkage baselines consistently outperform empirical-mean baselines, producing lower-variance gradient updates and improved training stability.

强化学习梯度优化大模型训练方差控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。