arXiv:2605.22074cs.LGcs.AI2026-05被引 1

通过分解推理链生成可验证子问题,提升大模型在难题上的学习效率。

From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning

论文配图:From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning
图 1 · 摘自论文原文
  • 从参考推理链中自动提取可验证子问题,构建课程学习框架。
  • 在7个数学推理基准上,平均准确率提升4.1点,难题探索能力显著增强。
  • 无需外部评分器,适合需要精细奖励分配的复杂逻辑推理任务。

基于可验证奖励的强化学习(RLVR)在大模型推理中展现出巨大潜力,但结果导向的RLVR在难题上效率低下,因为正确答案的回放稀少,且失败尝试中的部分进展无法获得奖励。本文提出SCRL(子问题课程强化学习),从参考推理链中生成可验证子问题,并将原始问题固定为最后一个子问题。这使得难题中的部分进展成为可验证的学习信号。算法上,SCRL采用子问题级归一化,独立地在每个子问题位置进行奖励归一化,并将得到的优势分配给对应的答案片段,实现无需外部评分标准或奖励模型的细粒度信用分配。分析表明,子问题课程能将难题从梯度死区中解放出来,且原问题越难,相对收益越大。在七个数学推理基准上,SCRL优于强基线,相比GRPO在Qwen3-4B-Base上平均准确率提升+4.1点,在Qwen3-14B-Base上提升+1.9点。在AIME24、AIME25和IMO-Bench上,SCRL在Qwen3-4B-Base上进一步将pass@1提升+3.7点,pass@64提升+4.6点,表明其在难题上的探索能力更强。

原文摘要 · Abstract (English)

Reinforcement learning from verifiable rewards (RLVR) has shown strong promise for LLM reasoning, but outcome-based RLVR remains inefficient on hard problems because correct final-answer rollouts are rare and sample-level credit assignment cannot use partial progress in failed attempts. We introduce SCRL (Subproblem Curriculum Reinforcement Learning), a curriculum RL framework that derives verifiable subproblems from reference reasoning chains and fixes the final subproblem as the original problem. This turns partial progress on hard problems into verifiable learning signals. Algorithmically, SCRL uses subproblem-level normalization, which normalizes rewards independently at each subproblem position and assigns the resulting advantages to the corresponding answer spans, enabling finer-grained credit assignment without external rubrics or reward models. Our analysis shows that subproblem curricula lift hard problems out of gradient dead zones, with larger relative gains as the original problem becomes harder. Across seven mathematical reasoning benchmarks, SCRL outperforms strong curriculum-learning baselines, improving average accuracy over GRPO by +4.1 points on Qwen3-4B-Base and +1.9 points on Qwen3-14B-Base. On AIME24, AIME25, and IMO-Bench, SCRL further improves pass@1 by +3.7 points and pass@64 by +4.6 points on Qwen3-4B-Base, indicating better exploration on hard reasoning problems.

强化学习推理链信用分配数学推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。