让教师先验证再教学，提升模型蒸馏的准确性和效率
Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation
- 按提示粒度检查教师可靠性，只在可信时才启用密集监督
- 在数学、代码等任务上，40亿和350亿参数学生均超越传统蒸馏
- 利用闲置教师算力做验证，大幅提高训练资源利用率
在策略蒸馏中，教师对学生的回放数据提供密集的词元级监督。传统方法对所有提示统一施加监督，未验证教师在每个提示上的可靠性。由于反KL损失具有模式聚焦特性，一个自信但错误的教师可能引发强烈而误导性的更新。现有分布代理（如熵或师生似然一致性）仅衡量不确定度或一致性，无法直接验证正确性。本文提出教师门控的策略蒸馏（TGOPD），核心思想是：在允许密集监督前，必须在提示层面验证教师可靠性。TGOPD通过少量经验证的教师探针估计可靠性，若通过则路由至密集蒸馏，否则使用验证器引导的GRPO。在40亿和350亿参数的学生模型上，跨数学、代码、指令跟随六项单领域任务中，TGOPD全面优于基线；多领域训练下，两类规模模型在七项基准平均表现均更高。通过将原本空闲的教师算力用于可靠性估计，异步蒸馏中教师节点显卡利用率从9.8%提升至78.9%。
原文摘要 · Abstract (English)
On-policy distillation (OPD) accelerates post-training by providing dense token-level supervision from a frozen teacher on the student's own rollouts. Vanilla OPD applies this supervision uniformly across prompts, without checking whether the teacher is reliable for each prompt. Because reverse KL is mode-seeking, a confidently wrong teacher can induce a strong yet misleading update. Distributional proxies, such as entropy or teacher-student likelihood agreement, measure uncertainty or agreement but do not directly verify outcome correctness. We introduce Teacher-Gated On-Policy Distillation (TGOPD), built on the principle that teacher reliability should be verified at the prompt level before dense supervision is admitted. TGOPD estimates reliability from a small set of verifier-scored teacher probes and routes each prompt exclusively to dense OPD when the reliability check passes or to verifier-grounded GRPO otherwise. Across 4B and 35B students in mathematics, code, and instruction following, TGOPD outperforms Vanilla OPD in all six single-domain settings and achieves higher seven-benchmark averages at both scales under multi-domain training. By using otherwise-idle teacher capacity for reliability estimation, TGOPD also reduces teacher-side compute waste in asynchronous OPD, increasing teacher-node GPU utilization from 9.8% to 78.9% in the measured 4B single-domain run.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。