改进强化学习中的推理模型，让其更可靠地泛化到新任务。
CoRPO: Adding a Correctness Bias to GRPO Improves Generalization
- 用正确性阈值裁剪基准，避免错误答案被误判为优秀。
- 在编码和数学任务上均超越GRPO，跨领域泛化能力更强。
- 适合需要稳定推理能力的复杂任务训练，如AI编程或数学证明。
组相对策略优化（GRPO）已成为通过强化学习训练大语言模型推理能力的标准方法。它通过组平均奖励估计优势，而非依赖学习的评判器，实现了可验证奖励强化学习（RLVR）的有效扩展。然而，我们发现一个根本性缺陷：GRPO的均值基线可能因劣质组平均表现，将错误解赋予正优势，导致优势高估并强化错误行为。为此，我们提出正确性相对策略优化（CoRPO），对GRPO目标进行简单修改，将最低基线裁剪至固定正确性阈值。我们证明，基线裁剪引入了保护性偏差，缓解过拟合同时保持有效探索。实验表明，使用CoRPO训练的模型在跨领域推理中表现更优，对域外（OOD）任务的泛化更一致。在编码任务上训练时，其数学性能优于GRPO；反之亦然，说明CoRPO学习的是鲁棒、可迁移的推理模式，而非特定任务解法。
原文摘要 · Abstract (English)
Group-Relative Policy Optimization (GRPO) has emerged as the standard for training reasoning capabilities in large language models through reinforcement learning. By estimating advantages using group-mean rewards rather than a learned critic, GRPO has enabled efficient scaling of reinforcement learning from verifiable rewards (RLVR). However, we identify a fundamental limitation: GRPO's mean baseline can assign positive advantages to incorrect solutions simply because they outperform a poorly-performing group average. It leads to overestimation of advantages and reinforcement of incorrect behaviours. To address this, we propose Correctness-Relative Policy Optimization (CoRPO), a simple modification to the GRPO objective that clips the minimum baseline to a fixed correctness threshold. We show that baseline clipping introduces a protective bias to advantage estimation that mitigates overfitting while preserving effective exploration. Empirically, CoRPO-trained models improve cross-domain reasoning, generalizing more consistently to out-of-domain (OOD) tasks. When trained on coding tasks, CoRPO outperforms GRPO on math, and vice-versa, indicating that CoRPO learns robust, transferable reasoning patterns rather than task-specific solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。