arXiv:2602.05165cs.LGcs.AI2026-02被引 4

提出EBPO方法,解决大模型强化学习中奖励不稳定问题。

EBPO: Empirical Bayes Shrinkage for Stabilizing Group-Relative Policy Optimization

  • 用经验贝叶斯收缩估计器融合局部与全局统计信息
  • 在小组规模下仍保持低方差和非消失梯度,提升训练稳定性
  • 适合需要稳定优化的大模型推理训练场景

基于可验证奖励的强化学习(RLVR)已证明能有效提升大语言模型(LLMs)的推理能力。然而主流方法如组相对策略优化(GRPO)面临严重稳定性挑战:在计算资源受限(小组规模小)时估计算法方差高,在所有响应均得零奖励的饱和失败场景下梯度信号消失。为此,我们提出经验贝叶斯策略优化(EBPO),通过借用策略累积的全局统计信息来正则化局部组基线。不同于孤立估计基线,EBPO采用动态平衡局部组统计与全局先验的收缩估计器,全局先验由Welford在线算法持续更新。理论上,我们证明了EBPO相比GRPO具有更低的均方误差(MSE)、有界熵衰减及失败场景下的非消失惩罚信号。实验上,EBPO在包括AIME和OlympiadBench在内的多个基准上持续优于GRPO及其他主流基线。尤其在小组规模较小时表现更优,且显著受益于难度分层课程学习,展现出卓越的训练稳定性。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for enhancing the reasoning capabilities of Large Language Models (LLMs). However, dominant approaches like Group Relative Policy Optimization (GRPO) face critical stability challenges: they suffer from high estimator variance under computational constraints (small group sizes) and vanishing gradient signals in saturated failure regimes where all responses yield identical zero rewards. To address this, we propose Empirical Bayes Policy Optimization (EBPO), a novel framework that regularizes local group-based baselines by borrowing strength from the policy's accumulated global statistics. Instead of estimating baselines in isolation, EBPO employs a shrinkage estimator that dynamically balances local group statistics with a global prior updated via Welford's online algorithm. Theoretically, we demonstrate that EBPO guarantees strictly lower Mean Squared Error (MSE), bounded entropy decay, and non-vanishing penalty signals in failure scenarios compared to GRPO. Empirically, EBPO consistently outperforms GRPO and other established baselines across diverse benchmarks, including AIME and OlympiadBench. Notably, EBPO exhibits superior training stability, achieving high-performance gains even with small group sizes, and benefits significantly from difficulty-stratified curriculum learning.

强化学习大模型推理稳定优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。