arXiv:2608.12957cs.LGcs.CL2026-08

根据回答正确性动态切换教学模式,提升大模型推理能力。

I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization

论文配图:I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization
图 1 · 摘自论文原文
  • 按实例判断是否启用教师指导,仅在全错时使用自蒸馏。
  • 在科学问答任务中平均准确率从56.67%提升至70.31%。
  • 自动降低教学依赖,无需人工设计调度策略。

组相对策略优化(GRPO)依赖采样结果间的奖励差异进行学习,但当所有响应均错误时无法获得有效相对信号。特权自蒸馏可提供密集的词元监督以填补这一空白,但全程应用会导致新问题:教师作为奖励目标的有偏、低方差近似,持续模仿会阻碍策略在具备成功轨迹能力后进一步优化。我们提出实例级自适应自蒸馏策略优化(I-SDPO),将教师依赖设为能力相关。I-SDPO对每个输入实例做一次路由决策,并共享至该实例的采样组:全错组采用特权自蒸馏目标,任意成功组则保留原GRPO机制。此设计仅在组内相对奖励无意义时使用模仿。局部分析揭示了教师方向与奖励方向对齐的条件,并表明非消失的有偏蒸馏权重会引入优化偏差下界。路由规则随成功率上升自动降低期望蒸馏率,无需人工调度即可撤回教师影响。在SciKnowEval上,I-SDPO在四个科学领域均取得最佳表现,平均mean@16准确率由GRPO的56.67%提升至70.31%,单领域最高提升达18.24分。

原文摘要 · Abstract (English)

Group Relative Policy Optimization (GRPO) learns from reward differences within a rollout group, but receives no useful relative signal when every sampled response is incorrect. Privileged self-distillation can fill this gap with dense token supervision, yet applying it throughout training creates a different failure mode: the teacher is a biased, low-variance surrogate for the reward objective, so persistent imitation can oppose reward-improving updates after the policy becomes capable of producing successful trajectories. We introduce I-SDPO (Instance-Level Adaptive Self-Distillation Policy Optimization), which treats teacher reliance as capability-dependent. I-SDPO makes one routing decision per input instance and shares it across that instance's rollout group: all-incorrect groups use a privileged self-distillation objective, whereas any-success groups remain intact for GRPO. This design uses imitation only where group-relative rewards are uninformative. A local analysis characterizes when teacher and reward directions align and shows that a non-vanishing biased distillation weight induces an optimization bias floor. The routing rule automatically reduces the expected distillation rate as success probability rises, withdrawing teacher influence without a hand-designed schedule. On SciKnowEval, I-SDPO obtains the best result in all four scientific domains and improves average mean@16 accuracy from 56.67% with GRPO to 70.31%, with a maximum domain gain of 18.24 points.

强化学习自蒸馏策略优化大模型推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。