arXiv:2603.13134cs.AI2026-03被引 11

通过对比正确与错误推理路径,提升大模型训练效果

When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO

  • 引入双向上下文机制,让模型在优化时对比成功与失败推理过程
  • 在数学推理任务上,多个模型和算法均实现稳定性能提升
  • 无需额外采样或辅助模型,适配所有GRPO变体,易部署

群体相对策略优化(GRPO)是训练推理模型的有效方法,其优势计算基于组内平均值。然而,GRPO在优化过程中将每个输出视为独立样本,忽略了同一组中正确与错误解法之间的天然对比结构,从而错失了可利用的丰富比较信息。本文揭示,GRPO目标实际上隐式最大化正确与错误样本间策略比值的间隔。基于此,提出双边上下文条件化(BICC)机制,使模型能在优化时交叉参考成功与失败的推理轨迹,实现样本间的信息直接流动。进一步提出奖励置信度校正(RCC),通过一阶近似方差最小估计器的奖励-置信度协方差动态调整优势基线,稳定训练过程。两项机制均无需额外采样或辅助模型,可适配所有GRPO变体。在数学推理基准测试中,实验显示多种模型与算法均取得持续提升。代码已开源。

原文摘要 · Abstract (English)

Group Relative Policy Optimization (GRPO) has emerged as an effective method for training reasoning models. While it computes advantages based on group mean, GRPO treats each output as an independent sample during the optimization and overlooks a vital structural signal: the natural contrast between correct and incorrect solutions within the same group, thus ignoring the rich, comparative data that could be leveraged by explicitly pitting successful reasoning traces against failed ones. To capitalize on this, we present a contrastive reformulation of GRPO, showing that the GRPO objective implicitly maximizes the margin between the policy ratios of correct and incorrect samples. Building on this insight, we propose Bilateral Context Conditioning (BICC), a mechanism that allows the model to cross-reference successful and failed reasoning traces during the optimization, enabling a direct information flow across samples. We further introduce Reward-Confidence Correction (RCC) to stabilize training by dynamically adjusts the advantage baseline in GRPO using reward-confidence covariance derived from the first-order approximation of the variance-minimizing estimator. Both mechanisms require no additional sampling or auxiliary models and can be adapted to all GRPO variants. Experiments on mathematical reasoning benchmarks demonstrate consistent improvements across comprehensive models and algorithms. Code is available at \href{https://github.com/Skylanding/BiCC}{https://github.com/Skylanding/BiCC}.

强化学习推理优化模型训练对比学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。