arXiv:2607.11506cs.LGcs.CL2026-07

通过分阶段奖励优化,让大模型推理更高效准确

SCOPE-RL: Optimizing Reasoning Paths Before and After Success

  • 前阶段用分解问题链生成中间奖励,后阶段用正确性门控修正路径
  • 在数学任务上准确率提升11.2个百分点,推理令牌减少27.1%
  • 适合想提升大模型推理质量与效率的研究者和开发者

基于可验证奖励的强化学习(RLVR)利用稀疏的最终答案奖励来优化大语言模型。这种稀疏锚点仅能验证轨迹是否成功,无法提供推理过程的直接反馈。成功前,难题的进展得不到奖励;成功后,结果奖励无法区分结构良好与冗余或局部错误的路径。我们提出SCOPE-RL(分层链路优化与过程效率),一个两阶段框架,在保留GRPO更新的前提下对奖励信号进行稠密化:自适应分层强化学习在成功前对隐藏答案的子问题链添加前缀分解的可验证奖励;质量感知过程强化学习在成功后对正确轨迹应用基于正确性的过程形状奖励。通过专家验证的步骤质量评估协议,评估有效步骤密度、错误定位能力与令牌效率。在基于DAPO-Math和Big-Math训练的Qwen3-8B-Instruct上,相比仅使用结果奖励的GRPO,SCOPE-RL平均准确率提升达11.2个百分点,推理令牌减少27.1%;该效果在GSPO下及Qwen3-0.6B-Instruct上依然成立,表明奖励信号稠密化与策略更新级改进具有互补性。代码与数据见https://github.com/tokencraft-lab/SCOPE-RL。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR) optimizes LLMs using sparse verifiable final-answer rewards. This sparse anchor reliably verifies whether a trajectory succeeds but provides no direct feedback on the reasoning path that produced it. Before success, prerequisite progress on hard problems receives no reward signal; after success, outcome rewards cannot distinguish well-organized correct trajectories from redundant or locally flawed ones. We introduce SCOPE-RL (Scaffolded Chain Optimization with Process Efficiency), a two-stage framework that densifies this anchor while retaining the GRPO update: Adaptive Scaffolded RL adds prefix-decomposed verifiable rewards on answer-hidden sub-question chains before success, and Quality-Aware Process RL applies correctness-gated process-shape rewards to refine correct trajectories after success. An expert-validated Step-Quality Evaluation Protocol evaluates useful-step density, error localization, and token efficiency beyond final-answer accuracy. On Qwen3-8B-Instruct trained on DAPO-Math and Big-Math, SCOPE-RL improves average accuracy by up to 11.2 pp and reduces reasoning tokens by up to 27.1% over outcome-only GRPO; the gains hold under GSPO and on Qwen3-0.6B-Instruct, indicating that reward-signal densification is complementary to policy-update-level RLVR advances. Code and data are available at https://github.com/tokencraft-lab/SCOPE-RL.

强化学习推理优化大模型奖励设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。