arXiv:2601.03525cs.LGcs.AI2026-01被引 1

用部分通过的测试用例生成密集奖励,提升代码生成强化学习效果

Beyond Binary: Turning Partial Success into Dense Verifiable Rewards for Reinforcement Learning in Code Generation

  • 基于测试用例通过情况构建可验证的密集奖励,避免传统方法的稀疏性
  • 解决简单测试用例主导训练的问题,提升模型在困难测试上的进步
  • 无需额外模型或显存开销,性能超越主流基线最高+8.83分

代码生成中的强化学习面临奖励设计的核心挑战。主流的测试集级结果奖励虽能保证功能正确性,但导致奖励稀疏;外部奖励模型虽提供密集监督,却存在对齐偏差和额外开销。由于代码评估天然产生多个测试用例级别的结果,部分成功(即通过部分测试)提供了内在且可验证的密集监督来源。本文提出VeRPO(可验证密集奖励策略优化),系统性地将可验证的部分成功转化为可靠密集奖励。我们通过加权求和形式分析部分成功奖励,理论发现一种关键的基数偏差,导致策略更新过度偏向简单测试的成功,而忽视前沿测试的进展。基于此,VeRPO引入动态密度校准的局部奖励,显式纠正该偏差,从部分成功中提供稳健的密集监督。为进一步对齐端到端功能正确性,VeRPO还将局部密集奖励与全局执行结果结合。在多种基准和设置下的实验表明,VeRPO优于基于结果和奖励模型的基线,实现最高+8.83的pass@1提升,时间成本低于0.02%,无额外GPU内存开销。

原文摘要 · Abstract (English)

Effective reward design is a central challenge in Reinforcement Learning (RL) for code generation. Mainstream test-suite-level outcome rewards enforce functional correctness but induce sparsity, while external Reward Models (RMs) provide dense supervision at the cost of misalignment and additional overhead. Since code evaluation naturally yields multiple test-case-level outcomes, partial success, i.e., passing a subset of test cases, offers an intrinsic, verifiable source of dense supervision. In this paper, we propose VeRPO (Verifiable Dense Reward Policy Optimization), an RL framework that systematically turns verifiable partial success into reliable dense rewards. We analyze partial-success rewards using a weighted sum formulation, theoretically identifying a critical cardinality bias that causes policy updates to disproportionately favor gains from easy-test successes over progress on frontier tests. Based on this, VeRPO introduces a dynamic, density-calibrated local reward that explicitly corrects this bias and provides robust dense supervision from partial success. To enhance alignment with end-to-end functional correctness, VeRPO further integrates the local dense reward with global execution outcomes. Extensive experiments across diverse benchmarks and settings demonstrate that VeRPO outperforms outcome-driven and RM-based baselines, achieving up to +8.83 pass@1 gain with negligible time cost (< 0.02%) and zero GPU memory overhead.

强化学习代码生成奖励设计密集奖励

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。