通过排序提示提升强化学习中教师监督的可靠性，显著改善模型训练效果。
ReOrder-OPD:Reliability-Aware Prompt Ordering for On-Policy Distillation
- 用代理指标衡量每个提示的教师延续可靠性，按可靠性高低排序提示。
- 在数学和代码任务上，相比随机或升序，降序排列使性能全面提升。
- 无需额外计算，仅靠一次学生采样即可实现高效可靠提示排序，适合部署到强化学习系统。
在策略蒸馏（OPD）中，教师对学生产生轨迹的逐标记监督并非始终可靠。现有方法依赖局部置信度或师生一致性来加权、过滤或截断采样轨迹，但这些信号无法直接判断教师是否能从学生前缀继续生成正确答案，且轨迹级干预可能混淆单一轨迹的不可靠性与提示的低训练价值。本文定义提示级教师延续可靠性 $R$ 为教师从学生前缀出发到达正确答案的概率，基于当前学生生成的前缀与轨迹进行平均。实验表明，高 $R$ 提示带来更大蒸馏增益，按 $R$ 降序训练优于随机与升序。由于精确估计 $R$ 需要大量教师延续，本文采用单次学生采样与验证器修正后的同提示教师轨迹间最大 ROUGE-5 F1 作为代理。在十个等频分箱中，该代理得分与真实 $R$ 呈单调上升关系，证明其可区分粗粒度可靠性水平。ReOrder-OPD 按此代理对提示排序，再进行独立采样训练。在 Qwen3 和 Gemma4 数学设置及 Qwen3 代码设置中，所有匹配聚合比较均取得改进；六组 FiRe-OPD 与 ExOPD 实验也显示提示排序与轨迹内监督具有互补性。
原文摘要 · Abstract (English)
On-policy distillation (OPD) applies token-level teacher supervision to student-generated trajectories, but this supervision is not always reliable. Existing methods use local confidence or teacher-student agreement to weight, filter, or truncate the sampled trajectory. These signals do not directly determine whether the teacher can continue a student prefix to a correct answer, and trajectory-level interventions can conflate one rollout's unreliability with low expected training value of its prompt. We define prompt-level teacher continuation reliability $R$ as the teacher's probability of reaching a correct answer from a student prefix, averaged over prefixes and trajectories induced by the current student. Oracle experiments show that high-$R$ prompts yield larger OPD gains and that descending-$R$ training outperforms random and ascending orders on a fixed prompt pool. Because estimating $R$ requires many teacher continuations, we use the maximum ROUGE-5 F1 between one independent student rollout and verifier-correct same-prompt teacher trajectories. Across ten equal-frequency bins of this actual score, mean $R$ rises monotonically, showing that the proxy separates coarse reliability levels. ReOrder-OPD sorts prompts by the proxy, then draws independent on-policy training trajectories for vanilla OPD. It improves every matched aggregate comparison across Qwen3 and Gemma4 mathematics settings and Qwen3 code settings. Gains in all six FiRe-OPD and ExOPD settings show that prompt ordering complements within-trajectory supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。