arXiv:2606.03532cs.LGcs.AI2026-06被引 3

提出新方法让教学模型更稳定,避免因更新时机不当导致失败。

When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation

论文配图:When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation
图 1 · 摘自论文原文
  • 通过冻结教师模型间隔提升训练稳定性,关键在隔离期而非教师年龄。
  • 发现长期训练中定时更新会引发不可逆错误,导致学习崩溃。
  • 设计自适应刷新机制,仅在学生真正进步时才更新教师模型。

自洽策略蒸馏中,学生策略基于自身参数历史生成教师模型,但教师更新频率这一时间耦合机制尚未被系统研究。在 Qwen3-8B 上的可控实验表明,教师完全冻结的隔离期是实现稳定学习的关键结构特性,而非教师年龄。为此提出诊断框架:时间KL结构、刷新冲击与长尾风险,揭示了‘状态无记忆坍塌’现象——最优短周期固定更新在长周期训练中会因时钟驱动的单步刷新,将瞬态漂移的学生状态复制进教师,造成不可逆失败。该失效模式在短周期评估下不可见,且与EMA的慢性污染机制不同。为此提出‘巩固门控教师刷新’(CGTR),在保留隔离期的同时,仅当奖励提升与长尾安全双重证据成立时才触发刷新,确保每次更新对应真实学习进展。使用单一参数集、无需每任务调参,CGTR在四个任务(化学、生物、物理、工具使用)中均实现零崩溃,并取得最佳最终得分,自动调节刷新频率以适配各任务学习动态。

原文摘要 · Abstract (English)

Self on-policy distillation trains a student policy against a teacher derived from its own parameter history, yet the teacher's update schedule -- which governs the \emph{temporal coupling} between teacher and student -- has not been systematically studied as a stability variable. Through a controlled schedule sweep on Qwen3-8B, we establish that \emph{isolation periods}, defined as complete teacher freezing between updates, are the key structural property enabling stable learning, not teacher age. To characterize these underlying training dynamics, we introduce a diagnostic framework of temporal KL structure, refresh shock, and length-tail risk. This framework further uncovers \emph{state-oblivious collapse}: optimal short-horizon fixed schedules catastrophically fail under long-horizon training because a clock-driven refresh can copy a transiently drifting student into the teacher in a single, irreversible step. This failure mode is invisible under short-horizon evaluation and mechanistically distinct from EMA's chronic contamination. To address this, we propose \emph{Consolidation-Gated Teacher Refresh} (CGTR), which preserves isolation periods while gating each refresh on joint evidence of reward improvement and length-tail safety, ensuring every teacher movement responds to genuine student consolidation rather than a clock signal. With a single shared parameter set and no per-dataset retuning, CGTR achieves \textbf{zero collapse} and the best final score on all four tasks (Chemistry, Biology, Physics, ToolUse), self-regulating its refresh frequency to each task's learning dynamics.

强化学习知识蒸馏训练稳定自洽学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。