用特权信息引导模型解决难题，提升推理能力
Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
- 训练时引入解题前缀等特权信息，生成非零奖励的推理路径
- 在数学推理基准上相比原版提升3.8%绝对性能
- 适合需要强推理能力的复杂任务研究者
基于可验证奖励的强化学习（RLVR）能提升大语言模型的推理能力。然而，传统方法在难题上表现不佳：当模型无法生成正确解时，会获得零学习信号。通过在训练中提供特权指导（如解题前缀），可引导模型走向有非零奖励的正确解。我们称这类从包含特权信息的提示生成的推演为“离上下文”（off-context）——其训练提示含指导，但目标仍基于原始无指导提示定义。本文提出离上下文GRPO（OC-GRPO），在GRPO基础上仅做最小修改，使用引导推演但通过重要性校正目标，使更新回归原始无指导目标，避免未校正引导训练导致的偏差。实验表明，该算法在标准数学推理基准上平均性能比原版GRPO提升3.8%（相对提升13.7%），额外计算开销可忽略。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) improves reasoning in large language models. Yet, typical RLVR approaches fail on difficult problems: when a model cannot generate any correct solutions, it receives \textit{zero} learning signal. Providing privileged guidance during training, such as solution prefixes, can help overcome this learning cliff by steering the model towards {correct solutions with non-zero reward}. {We call these rollouts \textit{off-context}: they are generated from a training prompt that contains privileged guidance, while the target objective is defined by the original prompt without that guidance.} {We introduce} Off-Context GRPO (OC-GRPO), a minimally modified variant of GRPO that uses guided rollouts but applies an importance-corrected objective to steer the update back toward the original unguided objective, avoiding the mismatch that destabilizes uncorrected guided training. Empirically, our algorithm achieves a 3.8\% absolute improvement (13.7\% relative gain) over vanilla GRPO on average across standard mathematical reasoning benchmarks with negligible additional cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。