通过分路自适应加权,让强化学习更精准地分配奖励信号。
SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting

- 根据轨迹正确性分流,分别用教师困惑度和学生困惑度加权
- 在六个推理基准上平均提升11.42%(Avg@32)和7.30%(Pass@32)
- 适合需要高精度奖励分配的大型语言模型对齐任务
基于策略的强化学习已成为大语言模型推理对齐的主流范式,但其稀疏的、结果级奖励使得词元级信用分配极为困难。基于策略蒸馏(OPD)通过引入教师模型的密集词元级KL监督缓解此问题,但通常对所有采样轨迹统一施加监督,忽略了信号质量的根本差异。本文提出信号校准的基于策略蒸馏增强方法(SCOPE),一种双路径自适应训练框架,根据轨迹正确性将策略采样分为两条互补的监督路径。对于错误轨迹,采用教师困惑度加权的KL蒸馏,优先选择教师具备真实纠错能力的样本,降低不可靠指导的影响;对于正确轨迹,则使用学生困惑度加权的最大似然估计,聚焦于能力边界上的低置信度样本,避免对已掌握内容过度强化。两条路径均采用群体级归一化,以适应不同提示的内在难度差异。在六个推理基准上的大量实验表明,与先进基线相比,SCOPE在Avg@32上平均提升11.42%,在Pass@32上提升7.30%,展现出持续有效性。
原文摘要 · Abstract (English)
On-policy reinforcement learning has become the dominant paradigm for reasoning alignment in large language models, yet its sparse, outcome-level rewards make token-level credit assignment notoriously difficult. On-Policy Distillation (OPD) alleviates this by introducing dense, token-level KL supervision from a teacher model, but typically applies this supervision uniformly across all rollouts, ignoring fundamental differences in signal quality. We propose Signal-Calibrated On-Policy Distillation Enhancement (SCOPE), a dual-path adaptive training framework that routes on-policy rollouts by correctness into two complementary supervision paths. For incorrect trajectories, SCOPE performs teacher-perplexity-weighted KL distillation to prioritize instances where the teacher demonstrates genuine corrective capability, while down-weighting unreliable guidance. For correct trajectories, it applies student-perplexity-weighted MLE to concentrate reinforcement on low-confidence samples at the capability boundary rather than over-reinforcing already mastered ones. Both paths employ a group-level normalization to adaptively calibrate weight distributions, accounting for the intrinsic difficulty variance across prompts. Extensive experiments on six reasoning benchmarks show that SCOPE achieves an average relative improvement of 11.42% in Avg@32 and 7.30% in Pass@32 over competitive baselines, demonstrating its consistent effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。