arXiv:2602.01523cs.LGcs.AI2026-02

提出相对预算理论,解释大模型推理中强化学习效率差异。

A Relative-Budget Theory for Reinforcement Learning with Verifiable Rewards in Large Language Model Reasoning

  • 用相对预算ξ=H/E[T]统一衡量采样效率,其中H为生成长度,T为首次正确解的词元数。
  • 实证发现ξ在[1.5,2.0]时学习效率最高,对应推理性能峰值。
  • 揭示三种学习状态:匮乏、平衡、冗余,提供可验证的在线强化学习保证。

强化学习是提升大语言模型推理能力的主要范式,但其效果在不同任务和计算预算下差异显著。本文提出一种相对预算理论,通过单一指标相对预算 ξ := H / E[T] 来解释这一现象,其中 H 为生成长度(词元预算),T 为基线策略下首次正确解出现所需的词元数。我们证明 ξ 决定样本效率,通过调控奖励方差和信息轨迹概率实现。分析揭示三种状态:在不足状态(ξ→0)中,信息轨迹稀少,样本复杂度爆炸;在平衡状态(ξ=Θ(1))中,信息轨迹以非可忽略概率出现,强化学习最高效;在充裕状态(ξ→∞)中,学习稳定但每轮边际收益递减。我们进一步提供在线强化学习的有限样本保证,刻画了不同状态下的学习进展。在理想分布假设下,案例研究显示相对预算随迭代线性增长。实证结果在真实设置中验证了这些预测,识别出 ξ ∈ [1.5, 2.0] 为学习效率最优区间,与推理性能峰值一致。

原文摘要 · Abstract (English)

Reinforcement learning (RL) is a dominant paradigm for improving the reasoning abilities of large language models, yet its effectiveness varies across tasks and compute budgets. We propose a \emph{relative-budget} theory explaining this variation through a single quantity called relative budget $ξ:= H/\mathbb{E}[T]$, where $H$ is the generation horizon (token budget) and $T$ denotes the number of tokens until the first correct solution under a base policy. We show that $ξ$ determines sample efficiency by controlling reward variance and the likelihood of informative trajectories. Our analysis reveals three regimes: in the \emph{deficient} regime ($ξ\to 0$), informative trajectories are rare and the sample complexity explodes; in the \emph{balanced} regime ($ξ=Θ(1)$), informative trajectories occur with non-negligible probability and RL is maximally sample-efficient; and in the \emph{ample} regime ($ξ\to \infty$), learning remains stable but marginal gains per iteration diminish. We further provide finite-sample guarantees for online RL that characterize learning progress across these regimes. Specifically, in a case study under idealized distributional assumptions, we show that the relative budget grows linearly over iterations. Our empirical results confirm these predictions in realistic settings, identifying a budget $ξ\in [1.5, 2.0]$ that maximizes learning efficiency and coincides with peak reasoning performance.

强化学习大模型推理样本效率相对预算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。