利用推理模型的正确与错误路径对比，自监督提升训练效率。
Self-Supervised On-Policy Distillation for Reasoning Language Models
- 从同一提示的多次尝试中提取正确与错误路径的对比信号，构建自监督蒸馏。
- 在三个数学竞赛数据集上，相比GRPO方法平均提升1.6分，最高达65.6分。
- 无需外部解题轨迹，适合需要高效推理训练的场景。
GRPO风格的强化学习验证与回复（RLVR）通过每个提示的多轮在线策略尝试训练推理模型，但通常仅利用最终奖励。我们发现，一组尝试中蕴含更丰富的过程信号：正确完成是当前策略解决问题的自生成证据，而错误完成则提供了策略需修正的在线前缀。本文提出自监督在线策略蒸馏（SSOPD），将基于最短正确完成的教师分布，蒸馏到最长错误完成的前缀上。该方法将组内正确-错误对比转化为密集过程监督，无需外部解题轨迹。停止时间视角表明，最短正确/最长错误规则是有限组近似于编辑持久失败以实现快速成功动作的策略；提示级前沿权重将辅助损失集中在正确与错误分支共存的位置。在AIME 2024、AIME 2025和HMMT 2025三个基准上，SSOPD在全部九个模型-基准组合中均优于GRPO。对于Qwen3-8B，在宏平均@12指标上达到65.6，比GRPO高1.6分，比解题条件下的OPSD基线高0.8分。代码将发布于https://github.com/tzq1999/SSOPD。
原文摘要 · Abstract (English)
GRPO-style RLVR trains reasoning models from multiple on-policy attempts per prompt, but typically uses these attempts only through terminal rewards. We show that a mixed group contains a richer process signal: a correct completion is a self-generated witness of how the current policy can solve the problem, while a wrong completion provides on-policy prefixes where the policy needs correction. We introduce \emph{Self-Supervised On-Policy Distillation} (SSOPD), which distills a teacher distribution conditioned on the shortest correct completion into prefixes of the longest wrong completion. This converts intra-group correct--wrong contrast into dense process supervision without external solution traces. A stopping-time view motivates the shortest-correct / longest-wrong rule as a finite-group approximation to editing persistent failures toward fast-success actions, and a prompt-level frontier weight concentrates the auxiliary loss where correct and wrong branches coexist. Across AIME 2024, AIME 2025, and HMMT 2025, SSOPD improves over GRPO in all nine model-benchmark settings. On Qwen3-8B, it reaches a macro Avg@12 of 65.6, outperforming GRPO by 1.6 points and the solution-conditioned OPSD baseline by 0.8 points. Code will be released at https://github.com/tzq1999/SSOPD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。