提出自适应奖励外推框架，让模型在模仿教师时更稳定可靠。
REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation
- 根据令牌可靠性动态调整奖励外推强度，避免过度拟合极端奖励。
- 在数学和代码任务中均优于现有方法，且无需额外模型或推理。
- 适合需要高稳定性与泛化能力的强化学习教学场景。
在线策略蒸馏（OPD）在密集令牌级监督下训练学生模型，以模仿教师生成的轨迹。如ExOPD等奖励外推方法通过放大教师参考对数似然比来超越直接模仿，但对所有令牌使用单一全局系数λ，导致学生可能过度拟合隐含奖励中的极端峰值，引发奖励黑客和训练不稳定，且最优λ值因领域而异，需耗费大量调参。本文提出REOPD：一种可靠性自适应的奖励外推框架。REOPD结合令牌级兼容性权重与批次级自适应预算，生成逐令牌系数λ_{b,t}=1+γ_b q_t，既能保持教师对齐，又能选择性地沿可靠教师方向外推。该方法无需验证器、奖励模型、价值模型或额外回放，仅依赖标准OPD流程。在单教师数学任务中优于G-OPD，多教师设置下在两个领域均表现更优；在单教师代码任务上与G-OPD持平，证明其在不同领域和教师配置下具备有效的细粒度可靠性适配能力。
原文摘要 · Abstract (English)
On-policy distillation (OPD) trains a student on its own trajectories under dense token-level supervision from a teacher. Reward-extrapolation methods such as ExOPD amplify the teacher-reference log-likelihood ratio to move beyond direct imitation, but apply a single global coefficient $λ$ to every token. This can drive the student to fit extreme peaks in the implicit reward, causing reward hacking and unstable training, and the optimal $λ$ varies across domains, requiring costly sweeps. We propose REOPD, a reliability-adaptive reward extrapolation framework for OPD. REOPD combines a token-level compatibility weight with a batch-level adaptive budget, yielding a token-wise coefficient $λ_{b,t}=1+γ_b q_t$ that preserves teacher alignment while selectively extrapolating along reliable teacher-reference directions. It requires no verifier, reward model, value model, or extra rollout beyond standard OPD. REOPD outperforms G-OPD on single-teacher mathematics and on both domains in the multi-teacher setting, while matching G-OPD on single-teacher code, demonstrating effective fine-grained reliability adaptation across domains and teacher configurations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。