arXiv:2605.11458cs.AIcs.CL2026-05被引 6

让大模型推理时的教师暴露程度自适应调整,提升学习效果。

Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning

论文配图:Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning
图 1 · 摘自论文原文
  • 用可学习的控制器动态调节教师看到的解题步骤量
  • 在多个数据集上相比原有方法提升0.95至2.33分
  • 适合想优化大模型推理能力的研究者和工程师

基于策略的自蒸馏已成为大模型推理的强大方法,其中教师指导学生自身生成的推理路径,并以参考答案为条件。然而几乎所有方法都默认教师始终可见完整参考推理过程,我们指出这一默认设定本身存在问题:当教师看到远超学生当前能力的推理内容时,产生的目标令牌过于强大,难以被学生吸收。控制实验表明,完全暴露并非总是最优,且师生能力差距随教师暴露增加而单调增长。为此,我们提出自适应教师暴露(ATESD),将暴露比例建模为轻量级贝塔策略控制器,依据训练状态统计信息动态调整,每次采样后维持短期学生更新窗口。为使控制器可学习,采用折扣化的学习进展奖励进行优化,通过评估决策对未来进步的影响而非即时损失变化来解决在线蒸馏带来的延迟信用分配问题。在AIME 24、AIME 25和HMMT 25三个数据集上,使用Qwen3-{1.7B, 4B, 8B}模型的实验显示,ATESD持续优于竞争性自蒸馏与强化学习基线,分别提升+0.95、+2.05、+2.33 Average@12分,确立自适应教师暴露是推理自蒸馏的有效新维度。

原文摘要 · Abstract (English)

On-policy self-distillation has become a strong recipe for LLM reasoning, where a privileged teacher supervises the student's own rollouts while conditioning on the reference solution. A design choice shared by nearly all such methods, however, has gone unquestioned: the teacher always sees the full reference reasoning. We argue that this default itself is part of the problem and identify a teacher-side exposure mismatch: when the teacher conditions on reasoning far beyond the student's current competence, the resulting token targets become too strong to absorb. A controlled fixed-exposure sweep makes this concrete on two fronts: 1) full exposure is not reliably the best choice, and 2) student-teacher mismatch grows monotonically as the teacher sees more privileged reasoning. This motivates treating teacher exposure not as a fixed hyperparameter but as a learnable training-time control variable. We therefore propose Adaptive Teacher Exposure for Self-Distillation (ATESD). ATESD models the reveal ratio with a lightweight Beta-policy controller conditioned on compact training-state statistics, and uses one sampled exposure for a short hold window of student updates. To make this exposure controller learnable, we optimize it with a discounted learning-progress reward that scores each held decision by its effect on the student's future improvement rather than its immediate loss change, addressing the delayed credit assignment induced by on-policy distillation. Experiments on AIME 24, AIME 25, and HMMT 25 across Qwen3-{1.7B, 4B, 8B} show that ATESD consistently outperforms competitive self-distillation and RL baselines, improving over OPSD by +0.95, +2.05, and +2.33 Average@12 points respectively, and establishing adaptive teacher exposure as an effective new axis for reasoning self-distillation.

大模型推理自蒸馏自适应控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。