解决强化学习中教师过度认同学生错误轨迹的问题
Escaping the KL Agreement Trap in On-Policy Distillation
- 引入动态阈值检测并终止低KL一致性陷阱
- 在4个数学基准上提升准确率2.66%~3.43%
- 适合追求高效强化学习训练的研究者
在线策略蒸馏（OPD）通过让教师对学生的轨迹进行评分，提供细粒度的令牌级监督。然而，当学生演变为不可恢复的前缀时，教师可能在局部与退化状态达成一致，产生低反向KL但缺乏修正信号。我们识别出这种持续存在的低KL一致性现象为‘低KL一致性陷阱’。进一步分析表明，陷阱期间及之后生成的令牌提供的监督信号效用较低。为此，我们提出KAT（KL一致性陷阱终止）机制，采用动态自适应阈值在线检测并终止此类陷阱。通过过滤劣质监督信号，KAT在四个数学基准上平均提升avg@k准确率2.66%、pass@k 3.43%，同时将平均轨迹长度减少59.73%。
原文摘要 · Abstract (English)
On-policy distillation (OPD) provides dense token-level supervision by asking a teacher to score student-generated rollouts. However, when the student drifts into an unrecoverable prefix, the teacher may locally agree with the degraded state, producing low reverse KL but little corrective training signal. We identify this persistent regime as a low-KL agreement trap. Further analyses show that tokens during and after such traps produce less useful supervision signals. We propose KAT (KL Agreement Trap Termination), an online OPD termination rule that detects persistent low-KL agreement with a dynamic training-adaptive threshold. By filtering weak supervision from degenerate agreement, KAT improves avg@k accuracy by 2.66% and pass@k by 3.43% across four mathematical benchmarks, while reducing average rollout length by 59.73%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。