arXiv:2606.00172cs.AI2026-06被引 2

提出无需参考答案的自教师强化学习方法,让模型更准地判断每步推理对错。

CAST: Non-Privileged Clipped Asymmetric Self-Teaching with Advantage Flipping for GRPO

论文配图:CAST: Non-Privileged Clipped Asymmetric Self-Teaching with Advantage Flipping for GRPO
图 1 · 摘自论文原文
  • 用停止梯度的自教师,根据轨迹正确性动态调整每步奖励信号
  • 在全对或全错情况下仍能提供有方向的反馈,避免梯度消失
  • 适合需要精准推理反馈的数学题等任务,尤其适合轻量级验证场景

基于可验证奖励的强化学习(RLVR),特别是群体相对策略优化(GRPO),已被广泛用于提升大语言模型的推理能力。然而,结果级奖励仅提供稀疏监督,当所有采样轨迹对错一致时,群体相对优势会消失。在策略自蒸馏(OPSD)中虽能提供密集的逐标记指导,但其标记偏好未必与轨迹正确性对齐;实证诊断显示,OPSD信号在正确与错误轨迹上表现不同,教师正负差距信号具有不同噪声特征。这些诊断仅在具备特权教师的分析场景下进行,而CAST训练采用无答案的自教师评分。受此启发,本文提出CAST:一种面向GRPO风格RLVR的无答案自蒸馏方法。它保留验证器驱动的GRPO目标,但使用停止梯度的自教师,依据轨迹正确性塑造令牌级优势。不同于以往方法,CAST无需依赖参考解答的教师评分,维持自教师概率差全程活跃,并引入双向局部优势符号翻转:正确轨迹中的教师负向标记可获负向令牌级优势,错误轨迹中的教师正向标记可获有界正向局部优势。对于零方差的全对或全错组,CAST赋予有界符号约束的基础优势,使原本梯度为零的组仍能获得验证器方向的令牌反馈。在数学推理任务上的实验表明,CAST在保持轻量、验证器导向的轨迹级目标的同时,提升了RLVR训练效果。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR), especially Group Relative Policy Optimization (GRPO), has been widely used to improve reasoning in large language models. However, outcome-level rewards provide only sparse supervision, and group-relative advantages vanish when all sampled trajectories for a prompt are either correct or incorrect. On-Policy Self-Distillation (OPSD) offers dense token-level guidance, but its token preferences are not necessarily aligned with trajectory correctness; empirical diagnostics show that OPSD signals behave differently on correct and incorrect rollouts, with teacher-positive and teacher-negative gap signals exhibiting different noise profiles. These diagnostics are conducted under an OPSD-style privileged teacher context for analysis only, whereas CAST training uses answer-free self-teacher scoring.Motivated by these observations, this work proposes CAST, an answer-free self-distillation method for GRPO-style RLVR. CAST keeps the verifier-grounded GRPO objective, but uses a stop-gradient self-teacher to shape token-level advantages according to trajectory correctness. Unlike prior self-distilled RLVR methods, CAST does not require reference-solution-conditioned teacher scoring, keeps the self-teacher log-probability gap active throughout training, and applies bidirectional local advantage sign reversal: teacher-negative tokens in correct trajectories can receive negative token-level advantages, while teacher-positive tokens in incorrect trajectories can receive bounded positive local advantages. For zero-variance all-correct and all-wrong groups, CAST assigns bounded sign-constrained base advantages, so these otherwise zero-gradient groups can contribute verifier-signed token feedback. Experiments on mathematical reasoning show that CAST improves RLVR training while retaining a lightweight, verifier-grounded trajectory-level objective.

强化学习推理增强自蒸馏验证器

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。