解决大策略差异下强化学习推理蒸馏的负反馈问题
Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
- 用教师模型指导学生生成每一步的文本，避免无效探索
- 在多个推理基准上优于传统方法，跨教师模型表现稳定
- 适合需要高鲁棒性推理蒸馏的大型语言模型应用
在线策略蒸馏（OPD）已成为面向推理的大型语言模型后训练的有前景范式，尤其与可验证奖励的强化学习（RLVR）结合时。现有OPD方法依赖于基于反向KL（RKL）的教师监督，对从学生策略采样的轨迹进行评估。然而，我们发现一个关键局限：当教师-学生策略差异较大时，强化学习驱动的探索常产生超出教师分布的轨迹，导致无信息量的负面反馈。为此，我们提出教师引导的策略优化（TGPO），一种在大策略差异下仍有效的在线策略推理蒸馏方法。TGPO不依赖单一评估监督，而是利用教师模型直接指导学生生成上下文相关的词元级输出；结合类似RLVR的轨迹级奖励，TGPO引导探索朝更优延续方向进行。在多个推理基准上的实验表明，TGPO持续优于现有的基于RKL的OPD方法，并在不同教师模型间保持鲁棒性。
原文摘要 · Abstract (English)
On-policy distillation (OPD) has become a promising paradigm for reasoning-oriented post-training of large language models (LLMs), especially when combined with reinforcement learning from verifiable rewards (RLVR). Existing OPD methods rely on reverse KL (RKL)-based teacher supervision over trajectories sampled from the student policy. However, we identify a critical limitation: under large teacher--student policy divergence, RL-driven exploration often produces trajectories outside the teacher distribution, resulting in uninformative negative feedback. To address this, we propose Teacher-Guided Policy Optimization (TGPO), an on-policy reasoning distillation method that remains effective under large policy divergence settings. Rather than relying solely on evaluative supervision, TGPO uses teacher to directly guide token level generation conditioning on student-generated contexts; together with RLVR-style trajectory level rewards, TGPO steers exploration toward improved continuations. Experiments on reasoning benchmarks show that TGPO consistently outperforms existing RKL-based OPD methods and remains robust across different teacher models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。