arXiv:2606.15455cs.LGcs.AI2026-06被引 7

揭示强化学习推理中多样性坍缩的根源,提出新方法提升模型推理能力边界。

Understanding Diversity Collapse in RLVR via the Lens of Overtraining

  • 从过训练视角解释多样性坍缩:成功后继续优化反而缩小推理范围。
  • 仅对未解问题更新可使高阶通过率超越基线模型,提升推理广度。
  • 提出贝叶斯边界门控机制,有效避免过训练,适用于复杂推理任务优化。

基于可验证奖励的强化学习(RLVR)已成为提升大语言模型推理能力的关键方法。然而,RLVR常出现多样性坍缩现象:Pass@$1$ 提升而高阶指标 Pass@$k$ 下降,表现为模型推理边界的收缩。本文从过训练视角剖析此现象:一旦问题对参考指标的贡献趋于饱和,后续更新不再拓展模型求解能力,却仍集中概率于策略采样偏好路径。在每题仅少量采样的标准设置下,单次成功即令问题进入近饱和状态,导致多数标准RLVR更新实质为过训练。该视角也提示:尽管高阶通过率整体下降,未必意味着无新推理能力产生。干预实验显示,仅对零成功问题更新,可在困难基准上使Pass@$256$ 超越基线;观察发现,部分初始不可解问题在训练中变为可解。基于此,本文提出贝叶斯边界门控(BBG),通过估计每个问题对推理边界的边际贡献,引导优化避开过训练区域。在多个推理基准上,BBG显著提升不同$k$值下的平均Pass@$k$。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR) has become a key approach for enhancing the reasoning abilities of large language models. However, RLVR often suffers from \emph{diversity collapse}: Pass@$1$ improves while high-$k$ Pass@$k$ degrades, which is viewed as a narrowing of the model's reasoning boundary. We formalize this diversity collapse through the lens of \emph{overtraining}: once a problem's contribution to the reference metric has effectively saturated, further updates no longer expand what the model can solve but still concentrate probability mass on the trajectories favored by on-policy sampling. Under a standard setup with few rollouts per problem, even a single observed success places a problem in a nearly saturated regime for high-$k$ Pass@$k$, so most updates in standard RLVR are overtraining from the boundary perspective. This perspective also suggests a reading of whether RLVR can expand the model's reasoning abilities beyond the base model: since RLVR is structurally biased against high-$k$ Pass@$k$, its aggregate decline does not by itself mean that no new reasoning gains occurred. Interventionally, restricting updates to problems with zero observed success lifts Pass@$256$ above the base model on difficult benchmarks; observationally, a non-trivial fraction of initially unsolvable problems become solvable during standard RLVR training. Building on these findings, we propose \emph{Bayesian Boundary Gating} (BBG), which redirects optimization away from overtraining by estimating each problem's marginal contribution to the reasoning boundary. Across multiple reasoning benchmarks, BBG improves average Pass@$k$ across a wide range of $k$.

强化学习推理能力多样性坍缩贝叶斯优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。