通过熵引导的自适应监督，让强化学习蒸馏更智能地分配注意力。
SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision
- 用熵值判断学生能力，动态选择需监督的文本片段
- 在六项数学任务上比传统方法平均提升4.8%准确率
- 适合追求高效强化学习蒸馏的研究者与工程师
在线策略蒸馏（OPD）存在一个离线蒸馏和强化学习中不存在的特性：教师监督质量依赖于学生的能力水平。能力不足时产生的不连贯轨迹会带来噪声梯度，而已掌握的标记则产生冗余梯度，造成三个层面的浪费（单个标记、训练阶段、提示）。现有方法对所有内容统一监督，效率低下。我们提出SEAD，利用熵作为统一探针，检测三尺度上的能力依赖性退化：(1) 联合教师-学生熵将标记划分为不同区域，仅对特定区域施加差异损失或置零梯度（约50%标记被跳过）；(2) 采用余弦调度，随能力增长从正向KL逐渐过渡到反向KL；(3) 采用基于能力的课程机制，按由易到难引入提示。这些组件相互依存：标记选择需要连贯轨迹（依赖课程），调度需单调提升（也依赖课程）。在OLMo-3（7B至32B）上，SEAD在六个数学基准上平均准确率较原始OPD提升4.8%，消融实验证明其交互具有超加性效应。
原文摘要 · Abstract (English)
On-policy distillation (OPD) has a property absent in offline distillation and RL: teacher supervision quality depends on student competence. Incoherent rollouts yield noisy gradients; already-mastered tokens yield redundant ones. This creates waste at three scales (tokens, training phases, and prompts) yet existing methods supervise uniformly. We introduce SEAD, which uses entropy as a unified probe of this competence-dependent degradation at three scales: (1) joint teacher-student entropy partitions tokens into zones receiving tailored divergences or zero gradient (approx. 50% skipped); (2) a cosine schedule anneals from forward to reverse KL as competence grows; (3) a competence-gated curriculum introduces prompts easy-to-hard. These components are symbiotically necessary: token selection requires coherent rollouts (curriculum), annealing requires monotonic improvement (also curriculum). On OLMo-3 (7B to 32B), SEAD achieves +4.8 avg accuracy over vanilla OPD across six math benchmarks, with ablations confirming super-additive interactions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。