解决大模型对齐中学生盲目模仿教师的虚假一致性问题。
Mismatch Matters: On-Policy Distillation Beyond Token Agreement
- 区分学生过量与缺失生成词，针对性修正不匹配问题。
- 在数学推理任务中将平均得分从6.9%提升至20.3%，响应长度缩短3.6倍。
- 无需采样缺失词即可恢复教师推理模式，适合强不对齐场景。
在线策略蒸馏（OPD）已成为现代大模型后训练的核心组件，但我们发现其存在一种失效模式：退化一致，即学生通过重复循环实现与教师近乎完美的词级一致，却产生全局错误响应。因此，我们从一致率转向关注师生不匹配，发现不匹配词主要分为两类：学生过量词（学生生成但教师赋予近零概率）和学生缺失词（教师偏好但学生极少采样）。前者导致对数比率校正无界并破坏更新，后者阻碍教师推理模式传递。为此，我们提出TIDE（Token-level Independent Deficit-Excess correction），通过有界Hellinger调节抑制最严重的过量生成，并采用解析式教师Top-K注入，在无需采样缺失词的前提下恢复概率质量。在多个Qwen3师生对的数学推理基准上，TIDE持续优于标准OPD及近期词选择与奖励调制基线。尤其在强师生不匹配下，其性能提升显著：平均得分从6.9%升至20.3%，平均响应长度减少3.6倍，格式错误大幅降低。代码已开源。
原文摘要 · Abstract (English)
On-policy distillation (OPD) has emerged as a core component of modern LLM post-training pipelines, yet we reveal a failure mode: degenerate agreement, where students exploit repetitive loops to achieve near-perfect token agreement with the teacher despite globally flawed responses. We therefore shift our focus from agreement to teacher-student mismatch, and find that mismatch tokens can be mainly categorized into two types: student-excess tokens and student-deficit tokens. Student-excess tokens are generated by the student but assigned near-zero probability by the teacher; their log-ratio corrections grow unbounded and destabilize the update. Student-deficit tokens, in contrast, are preferred by the teacher but rarely sampled by the student; their absence blocks the transfer of the teacher's reasoning patterns. To tackle these mismatch directions, we propose TIDE (Token-level Independent Deficit-Excess correction), which applies bounded Hellinger shaping to suppress the most severe sampled excesses and an analytic teacher top-$K$ injection to restore deficient probability mass without requiring deficit tokens to be sampled. Across mathematical reasoning benchmarks with multiple Qwen3 teacher-student pairs, TIDE consistently outperforms standard OPD and recent token-selection and reward-shaping baselines. Moreover, the gains of TIDE are more pronounced under strong teacher-student mismatch, where it improves Avg@8 from 6.9% to 20.3%, reduces average response length by a factor of 3.6, and substantially reduces formatting failures. Code is available at https://github.com/yzc-666/TIDE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。