VISTA让教师根据学生表现动态调整,提升推理模型自蒸馏效果。
VISTA: Verifier-Informed Student-to-Teacher Adaptation for On-Policy Self-Distillation

- 用验证通过的回放数据反向优化教师模型,使其更贴合学生推理路径
- 在三个基准上,相比标准自蒸馏分别提升0.6、0.7和2.1分
- 无需额外采样或奖励机制,适合想提升推理能力的模型开发者
在策略自蒸馏(OPSD)中,学生模型通过教师提供的密集令牌级监督进行训练,但传统方法将教师分布视为固定目标,忽略了教师可能偏离学生合理推理路径的问题。为此,本文提出验证器引导的学生到教师适应方法(VISTA),在保持标准学生更新的同时,利用结果验证过的回放序列来调整教师模型,使其趋向学生分布。对于每个验证过的回放,仅对教师与学生间KL散度最大的前k个位置进行适应。值得注意的是,VISTA复用原始回放和损失函数,不引入额外采样或独立奖励目标。在AIME24、AIME25和HMMT25三个数据集上,使用Qwen3系列1.7B、4B、8B模型进行实验,VISTA在所有规模下均取得最高平均精度@12,分别较标准OPSD提升0.6、0.7和2.1分。结果表明,基于验证结果的学生监督具有显著价值,且学生引导教师适应是提升自蒸馏的有效方向。
原文摘要 · Abstract (English)
On-policy self-distillation (OPSD) improves reasoning by training a problem-only student on its own rollouts using dense token-level supervision from a privileged teacher that also sees a reference solution. However, standard OPSD treats the teacher distribution as a fixed target along the student's rollout and updates only the student %, although -- even though privileged conditioning does not guarantee that the teacher always provides the most appropriate target for problem-only reasoning. This one-way supervision can therefore misdirect the student when the teacher distribution is misaligned with valid student reasoning. We therefore introduce Verifier-Informed Student-to-Teacher Adaptation (VISTA), which preserves the standard OPSD student update while using outcome-verified rollouts to adapt the teacher toward the student distribution. Within each verified rollout, VISTA further restricts this adaptation to the top-$k$ positions with the largest teacher--student KL divergence. Notably, VISTA reuses the rollout and loss function from standard OPSD, introducing no additional sampling or separate reward objective. Across AIME24, AIME25, and HMMT25 with Qwen3 models at 1.7B, 4B, and 8B, VISTA achieves the highest Avg@12 at every scale, improving over OPSD by $0.6$, $0.7$, and $2.1$ points, respectively. These results demonstrate the value of student supervision from outcome-verified rollouts and highlight student-to-teacher adaptation as a promising direction for OPSD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。