通过课程式强化学习突破大模型推理能力边界,提升多步推理效果。
Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model

- 设计边界感知课程强化学习,主动探索并引导模型突破现有推理瓶颈。
- 在多个基底模型上,pass@256平均提升9.8个百分点,显著超越基础模型与普通RLVR。
- 适合希望持续提升大模型复杂推理能力的研究者和开发者使用。
基于可验证奖励的强化学习(RLVR)被视为持续改进大语言模型的有前景路径。然而,近期研究表明,主流RLVR通常仅在基底模型已有的轨迹间重新分配采样概率:虽能提升采样效率(表现为更高的pass@1得分),但增益有限,且当k较大时反而可能降低pass@k得分,难以拓展基底模型的推理能力边界。本文提出一种边界感知课程强化学习方法,首先利用pass@k采样定位当前推理能力边界,再对边界附近或超出边界的样本施加定向教师引导,并最终通过强化学习固化新引入的推理模式。在Qwen、Llama和DeepSeek等基底模型上,该方法同时提升了pass@1与pass@256得分,其中pass@1反映单次尝试性能,pass@256作为推理能力边界的实证代理。实验显示,平均pass@256相比基底模型提升9.8个百分点,相比普通RLVR提升10.3个百分点。结果表明,边界感知课程强化学习为大模型持续突破基底模型的实证推理能力边界提供了可扩展的路径。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) is widely viewed as a promising path toward continuously improving large language models. Recent works, however, suggest that mainstream RLVR often reallocates sampling probabilities among trajectories already present in the base model: it can improve sampling efficiency, reflected by higher pass@1 scores, but yields limited gains, and can even decrease pass@k scores when k is large, and therefore may fail to expand the base model's reasoning capacity boundary. In this paper, we present a boundary-aware Curriculum RL approach to move beyond the base model's reasoning capacity boundary. Our approach first uses pass@k sampling to locate the current reasoning capacity boundary, then applies targeted teacher guidance to examples near or beyond that boundary, and finally uses RL to consolidate the newly introduced reasoning patterns. Across Qwen, Llama, and DeepSeek base models, boundary-aware Curriculum RL improves both pass@1 scores and pass@256 scores, with pass@1 reflecting one-attempt performance and pass@256 serving as an empirical proxy for the reasoning capacity boundary. In our experiments, average pass@256 improves by 9.8 percentage points over the base models and by 10.3 percentage points over Vanilla RLVR. These results suggest that boundary-aware Curriculum RL can provide a scalable route for LLMs to continuously improve beyond the base model's empirical reasoning capacity boundary.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。