解决自奖励强化学习中的系统性偏差问题，提升模型稳定性与性能。
Breaking the Self-Confirming Loop: Diagnosing and Mitigating Systemic Reward Bias in Self-Rewarding RL
- 通过多模型集成与动态奖励融合，降低自我奖励的耦合偏差。
- 实验显示性能提升6.2%，接近有标注数据的强化学习效果。
- 适合研究大模型自监督训练与奖励机制设计的学者参考。
基于可验证奖励的强化学习（RLVR）虽能有效扩展大语言模型的推理能力，但受限于标注数据稀缺。自奖励强化学习（RLIR）虽具可扩展性，却常因不稳定性与性能较差而受限。本文揭示其根源在于信心耦合的自奖励存在系统性偏差：模型倾向于过度奖励高置信度的错误，形成自我强化循环。我们提出三个量化指标：奖励噪声幅度（rho_noise）、策略-奖励耦合度（rho_selfbias）及过/欠奖励偏斜度（rho_symbias）。分析表明，强耦合会放大置信度相关的误差，导致奖励过度漂移，引发不稳定并限制性能上限。为此，提出基于集成奖励的强化学习（RLER），通过自适应奖励插值与分歧感知的轨迹选择，降低耦合性并抑制过奖励漂移。大量实验表明，RLER相比最佳RLIR基线提升6.2%，性能仅比RLVR低3.6%，且在无标注样本上表现出稳定增长。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) efficiently scales the reasoning ability of large language models (LLMs) but is bottlenecked by scarce labeled data. Reinforcement learning with intrinsic rewards (RLIR) offers a scalable alternative via self-rewarding, yet often suffers from instability and inferior performance. We trace this gap to a systemic bias in confidence-coupled self-rewarding: the model tends to over-reward high-confidence mistakes, forming a self-confirming loop. We quantify this feedback-loop bias with three metrics: reward noise magnitude (rho_noise), policy-reward coupling (rho_selfbias), and over-/under-reward skew (rho_symbias). Our analyses show a compounding effect where strong coupling amplifies confidence-conditioned errors and drives a drift toward over-reward, leading to instability and a lower performance ceiling. To mitigate this, we propose reinforcement learning with ensembled rewards (RLER), which aggregates diverse models with adaptive reward interpolation and disagreement-aware rollout selection to reduce coupling and suppress over-reward drift. Extensive experiments show that RLER improves by 6.2% over the best RLIR baseline and is within 3.6% of RLVR, while exhibiting stable scaling on unlabeled samples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。