让强化学习在失败样本中仍能有效学习，提升模型推理能力。
Selective Off-Policy Reference Tuning with Plan Guidance
- 从参考解中提取计划，动态调整失败样本的权重
- 在8个推理基准上优于GRPO，弱模型提升更明显
- 适合需要增强推理鲁棒性的模型优化场景
带有可验证奖励的强化学习有助于提升推理能力，但GRPO类方法在所有采样轨迹均失败的难题上会停滞。SORT通过修复更新解决此问题：从参考解中提取计划，比较有无该计划时的词元概率，对在计划条件下变得更可预测的词元赋予更高权重。这使全错提示转化为选择性、结构感知的学习信号，而非均匀模仿。在三个骨干网络和八个推理基准上，SORT均优于GRPO和引导基线，尤其在较弱模型上提升显著。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards helps reasoning, but GRPO-style methods stall on hard prompts where all sampled rollouts fail. SORT adds a repair update for those failures without changing rollout generation: it derives a plan from the reference solution, compares token probabilities with and without that plan, and gives higher weight to tokens that become more predictable under plan conditioning. This turns all-wrong prompts into selective, structure-aware learning signals instead of uniform imitation. Across three backbones and eight reasoning benchmarks, SORT improves over GRPO and guidance baselines, with largest gains on weaker models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。