arXiv:2608.24696cs.LGcs.AI2026-08

将轨迹正确性融入强化学习蒸馏,提升大模型推理能力。

On-policy Distillation with Verifiable Reward

论文配图:On-policy Distillation with Verifiable Reward
图 1 · 摘自论文原文
  • 基于轨迹正确性重构采样令牌的隐式奖励信号
  • 通过ReLU门控机制实现正确轨迹得正分、错误轨迹得负分
  • 无需额外超参,可无缝对接各类策略梯度算法

基于可验证奖励的强化学习(RLVR)与在线策略蒸馏(OPD)已成为大语言模型后训练的两种主流范式。然而,RLVR存在任务层面反馈稀疏的问题,而OPD虽提供密集的令牌级指导,却忽略轨迹正确性,导致性能受限于教师模型。现有融合方法多依赖加权组合或启发式切换,引入额外超参数和权衡。本文提出无超参的在线策略蒸馏结合可验证奖励(OPDVR),首先根据轨迹正确性重构采样令牌的隐式奖励,再引入ReLU门控机制,确保正确轨迹获得非负奖励,错误轨迹获得非正奖励,从而对齐蒸馏信号与任务成功目标,同时保留教师分布引导。此外,该修改使采样令牌型OPD转化为有效的RLVR方法,可直接兼容任意策略梯度算法(如GRPO)。在六个推理基准上的实验表明,OPDVR始终优于标准OPD。代码已开源。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training large language models. However, RLVR suffers from sparse task-level feedback, while OPD provides dense token-level guidance but ignores trajectory correctness, limiting its performance to that of the teacher. Combining them is a promising direction: OPD supplies dense supervisory signals, while RLVR provides task-level correctness. Nevertheless, existing integrations often rely on weighted combination or heuristic switching, introducing extra hyperparameters and trade-offs. We propose On-policy Distillation with Verifiable Reward (OPDVR), a simple yet effective method that seamlessly combines OPD and RLVR without adding any hyperparameters. We first reformulate the implicit reward of sampled-token OPD based on trajectory correctness, then apply a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards---thereby aligning the distillation signal with task success while preserving the teacher's distributional guidance. Furthermore, our modification transforms sampled-token OPD into a proper RLVR method, making it readily combinable with any policy gradient algorithm, such as GRPO. Experiments on six reasoning benchmarks show that OPDVR consistently outperforms standard OPD. Our code is available at https://github.com/LeapLabTHU/OPDVR.

强化学习模型蒸馏大模型推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。