通过分块策略漂移门控提升强化学习中学生模型的推理稳定性。
Blockwise Policy-Drift Gating for On-Policy Distillation
- 基于学生模型自身行为与当前策略的分布差异，分块计算漂移信号。
- 在数学推理任务上，平均通过率从49.78%提升至51.60%。
- 无需修改教师模型，适合追求鲁棒性的强化学习研究者使用。
在线策略蒸馏（OPD）利用学生模型采样轨迹上的教师信号进行训练。近期研究表明，采样标记的OPD在长时序推理任务中易受干扰，而局部教师支持匹配可有效缓解。本文提出一种轻量级的学生端旧-当前策略漂移门控机制，适用于回放重用场景。该方法在采样路径上计算行为学生与当前学生之间的对数概率偏移，将偏移量按固定块聚合，并生成去关联、均值归一化的门控信号，用于重新加权OPD位置损失。该方法不改变教师目标、教师前K支持或回放策略。在六种变体的Qwen3数学推理基准测试中，统一设置200步训练预算，以pass@8作为核心解题率指标。固定64标记块门控使OPD在AIME24、AIME25、MATH500和AMC23四个数据集上的平均pass@8从0.4978提升至0.5160。在Teacher-TopK/LSM设置下，Block64在所有训练学生中取得四基准平均最佳pass@8表现。结果表明，局部旧-当前策略漂移是重用回放样本中可操作的控制信号，建议将分块门控作为提升解题率鲁棒性的默认策略。
原文摘要 · Abstract (English)
On-policy distillation (OPD) trains a student policy using teacher signals computed on trajectories sampled by the student itself. Recent work shows that sampled-token OPD can be fragile on long-horizon reasoning tasks and that local teacher-support matching is a simple and effective repair. This paper introduces blockwise policy-drift gating, a lightweight student-only old-current drift controller for OPD under rollout reuse. The method computes log-probability shifts between the behavior student and the current student on the sampled token path, aggregates these shifts over fixed blocks or spans, and uses the resulting detached, mean-normalized gates to reweight OPD position losses. It does not change teacher targets, teacher top-K supports, or the rollout policy. In a six-variant Qwen3 math reasoning benchmark with a uniform 200-step training budget for all trained variants, we use pass@8 as the primary problem-level solve-rate metric. Fixed 64-token block gating improves sampled-token OPD mean pass@8 from 0.4978 to 0.5160 across AIME24, AIME25, MATH500, and AMC23. On Teacher-TopK/LSM, Block64 gives the best four-benchmark mean pass@8 among trained students. The results identify local old-current policy drift as a practical control signal for reused OPD rollouts and motivate block-level gating as a simple default for improving solve-rate robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。