SPOT让学生模型更聪明地学老师,提升推理能力。
SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation
- 用三重机制决定哪里探查、探什么,精准分配学习资源。
- 在多个推理任务上显著提升学生模型表现,兼顾解的质量与覆盖度。
- 适合想改进强化学习中策略蒸馏效果的研究者使用。
在线策略蒸馏(OPD)为学生生成的轨迹提供密集的教师监督,但标准反KL训练可能对其他合理后续动作赋予过低概率。仅看教师熵无法判断不确定性是集中在少数合理选项上,还是分散在长尾分布中,也无法判断学生是否已良好表示这些候选。此外,局部教师概率未必能预测下游成功。我们提出稀疏探测与结果校准目标的OPD(SPOT),通过获取-探索-利用过程解决两个耦合决策:何时探查、何内容蒸馏。在获取阶段,位置级评分结合归一化教师熵、小顶-k候选集的概率质量及学生-教师差异,分配有限探查预算;在探索阶段,通过验证器评分的学生延续评估教师提议的候选;在利用阶段,这些结果生成闭式KL正则化目标,偏好具有更好下游表现的候选,同时保持与教师分布一致。在多个学生模型和推理基准上的大量实验表明,SPOT有效提升推理性能,同时平衡解的质量与覆盖范围。
原文摘要 · Abstract (English)
On-policy distillation (OPD) provides dense teacher supervision on student-generated trajectories, but standard reverse-KL training can assign insufficient probability to other plausible continuations. Teacher entropy alone does not reveal whether uncertainty is concentrated among a few plausible next tokens or dispersed over a long probability tail, nor whether the student already represents those candidates well. Moreover, local teacher probabilities may not predict downstream success. We introduce Sparse Probing and Outcome-calibrated Targets OPD (SPOT), which addresses two coupled decisions, where to probe and what to distill, through an acquisition--exploration--exploitation procedure. During acquisition, a position-level score combines normalized teacher entropy, the probability mass captured by a small top-$k$ candidate set, and student--teacher mismatch to allocate a limited probing budget. During exploration, SPOT evaluates teacher-proposed candidates through verifier-scored student continuations. During exploitation, these outcomes produce a closed-form, KL-regularized target that favors candidates with better downstream outcomes while remaining anchored to the teacher distribution. Extensive experiments across multiple student models and reasoning benchmarks demonstrate the effectiveness of SPOT in improving reasoning performance while balancing solution quality and coverage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。