提出可教性评估机制，精准筛选真正可学的提示词进行强化学习蒸馏。
Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
- 基于局部兼容性定义'可教性'，区分可纠正与不可纠正的教师信号。
- 仅用5%高可教性位置蒸馏，性能超越全量蒸馏和现有基线。
- 无需奖励模型或验证器，轻量高效，适合大模型对齐优化场景。
在策略蒸馏（OPD）中，学生模型通过自身生成的轨迹接受逐标记的教师指导。近期方法通过优先选择高熵或高分歧标记来提升效果，但未考虑这些信号是否真正可学。本文提出固定上下文诊断法，测量教师-学生间KL散度的减少量，发现原始KL分歧是粗略代理，混淆了可纠正的分歧（教师将修正概率分配给学生前K个候选）与不可兼容的分歧（教师概率集中在学生支持集之外）。为此，作者形式化提出'标记可教性'概念，证明其比原始分歧更能预测蒸馏改进效果。据此设计了轻量级的可教性感知蒸馏方法（TA-OPD），仅对高可教性标记应用蒸馏损失，无需奖励模型或验证器。在Qwen2.5与Qwen3师生设置下，TA-OPD仅保留5%标记即常优于全量蒸馏，并显著超过基于熵和发散的基线。结果表明，选择性蒸馏应聚焦于真正可学的教师信号，而非仅关注显著性。
原文摘要 · Abstract (English)
On-policy distillation (OPD) trains a student on its own rollouts with token-level teacher supervision. Recent selective OPD methods exploit the non-uniformity of OPD signals by prioritizing high-entropy or high-disagreement tokens. We revisit this principle and ask: which token-level teacher signals are actually learnable? Using a fixed-context diagnostic that measures same-context teacher-student KL reduction, we show that raw KL disagreement is a coarse proxy for learning value. It conflates learnable disagreement, where the teacher assigns corrective mass to the student's top-K candidates, with incompatible disagreement, where the teacher places mass mostly off the student's current support. We formalize this local compatibility as token teachability and show that it better predicts fixed-context improvement than raw KL alone. Motivated by this finding, we propose Teachability-Aware OPD (TA-OPD), a lightweight token-position selection method that applies OPD loss to high-teachability positions without reward models or verifiers. Across Qwen2.5 and Qwen 3 teacher-student settings, TA-OPD often surpasses full-token OPD with only 5% retained tokens and improves over entropy- and divergence-based baselines. Our results reframe selective OPD as selecting learnable teacher signals rather than merely salient tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。