arXiv:2608.09447cs.LGcs.AI2026-08

通过双策略协作训练,提升强化学习中学生模型的稳定性与性能。

WDL-OPD: Weak-Driven On-Policy Distillation via Mixture-Constrained Co-Training

  • 引入两个可训练策略,通过混合分布匹配教师模型,稳定训练过程。
  • 在1.7B和4B模型上,数学任务准确率分别提升至0.585和0.685。
  • 适合追求强化学习训练稳定性的研究者与工程实践者。

在线策略蒸馏(OPD)通过学生自身采样的轨迹对齐师生策略,缓解离线蒸馏的训练-测试状态不一致问题。然而,同一反馈环路可能引发不稳定:每次更新同时改变策略与后续更新所依赖的状态。本文提出WDL-OPD,一种基于混合约束的协同训练方法,包含一个锚定策略生成轨迹,一个辅助策略评估相同状态,两者令牌分布的几何混合通过反向KL散度匹配冻结的教师模型。两个策略均接收梯度。我们证明,冻结辅助策略可恢复近似于OPD²和W2S-OPD的锚点加对比代理目标;而联合训练则引入分支级自由度,是静态偏移无法表达的。在1.7B和4B规模的Qwen3实验中,WDL-OPD在四个尺度-领域设置下均生成最强学生检查点。其在4B模型上将MATH500准确率从0.630提升至0.685,在1.7B模型上从0.521提升至0.585。代码生成任务中,七种单策略OPD配置出现熵增长或轨迹退化,而协同训练达独立重评开发得分0.637和0.375。由于部分对比存在课程或初始化差异,结果支持稳定性假设而非普遍因果结论。本文提供精确训练算法、失败证据与受控比较矩阵以验证该假设。

原文摘要 · Abstract (English)

On-policy distillation (OPD) aligns a student with a teacher on trajectories sampled from the student itself, reducing the train-test state mismatch of offline distillation. The same feedback loop can nevertheless be unstable: each update changes both the policy and the states on which the next update is computed. We introduce WDL-OPD, a mixture-constrained co-training method with two trainable policies. An anchor policy generates every rollout, an auxiliary policy evaluates the same visited states, and a geometric mixture of their token distributions is matched to a frozen teacher by reverse KL. Both policies receive gradient. We show that freezing the auxiliary recovers an anchor-plus-contrast proxy target closely related to OPD$^2$ and W2S-OPD, whereas joint training creates branch-level degrees of freedom that a static delta cannot express. In recorded Qwen3 experiments at 1.7B and 4B scale, WDL-OPD produces the strongest student checkpoint in each of four scale-domain settings. It raises MATH500 accuracy from 0.630 to 0.685 at 4B and from 0.521 to 0.585 at 1.7B. In code generation, seven single-policy OPD configurations exhibit entropy growth or trajectory degradation, while co-training reaches independently re-evaluated development scores of 0.637 and 0.375. Because several comparisons differ in curriculum or initialization, these results support a stabilization hypothesis rather than a universal causal claim. We provide the exact training algorithm, failure evidence, and the controlled comparison matrix needed to test that hypothesis.

强化学习模型蒸馏协同训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。