利用多次尝试中的成功与失败样本,提升大模型推理的监督效果。
Multi-Rollout On-Policy Distillation via Peer Successes and Failures

- 通过同伴样本构建教师信号,融合成功与失败信息进行联合监督。
- 在编程、数学等任务上,相比传统方法提升12%~18%性能。
- 适合需要多轮试错优化的复杂推理场景,如竞赛编程与科学问答。
大语言模型常通过稀疏验证器奖励进行后训练,仅指示采样轨迹是否成功,难以提供具体推理成败位置的指导。现有在线策略蒸馏(OPD)方法通常独立处理每条轨迹,忽略同一提示下的其他尝试。本文提出多轮次在线策略蒸馏(MOPD),一种基于同伴行为的蒸馏框架,利用学生生成的局部轨迹组构建更丰富的教师信号。MOPD同时以成功和失败的同伴轨迹为条件:成功轨迹提供有效推理模式的正向证据,失败轨迹则提供应避免的常见错误结构化负向证据。我们研究了两种同伴上下文构造方式:正向模仿与正负对比。在编程竞赛、数学推理、科学问答及工具使用基准测试中,MOPD持续优于标准在线策略基线。教师信号分析显示,混合成功-失败上下文使教师评分更贴近验证器奖励,表明性能提升源于更忠实、实例自适应的监督。结果表明,有效的在线蒸馏应利用学生多轮试错行为,而非将轨迹视为孤立样本。
原文摘要 · Abstract (English)
Large language models are often post-trained with sparse verifier rewards, which indicate whether a sampled trajectory succeeds but provide limited guidance about where reasoning succeeds or fails. On-policy distillation (OPD) offers denser token-level supervision by training on student-generated trajectories, yet existing methods typically distill each rollout independently and ignore the other attempts sampled for the same prompt. We introduce Multi-Rollout On-Policy Distillation (MOPD), a peer-conditioned distillation framework that uses the student's local rollout group to construct more informative teacher signals. MOPD conditions the teacher on both successful and failed peer rollouts: successes provide positive evidence for valid reasoning patterns, while failures provide structured negative evidence about plausible mistakes to avoid. We study two peer-context constructions: positive peer imitation and contrastive success-failure conditioning. Experiments on competitive programming, mathematical reasoning, scientific question answering, and tool-use benchmarks show that MOPD consistently improves over standard on-policy baselines. Further teacher-signal analysis shows that mixed success-failure contexts better align teacher scores with verifier rewards, indicating that the gains arise from more faithful, instance-adaptive supervision. These results indicate that effective on-policy distillation should exploit the student's multi-rollout trial-and-error behavior rather than treating rollouts as isolated samples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。