用推理过程奖励提升代码生成质量,解决奖励欺骗问题。
ReCode: Reinforcing Code Generation with Reasoning-Process Rewards
- 通过对比优化与退化的推理路径训练奖励模型,评估推理质量。
- 7B模型在HumanEval+上比基础版提升16.1%,接近GPT-4-Turbo水平。
- 适合关注代码生成与强化学习结合的研究者,尤其重视推理过程的场景。
实际中,严谨的推理常是正确代码的关键,但现有代码生成的强化学习方法往往忽略推理质量的优化。将过程级监督引入强化学习虽具吸引力,却面临两大挑战:其一,高质量推理评估的奖励模型训练受限于细粒度偏好数据稀缺;其二,直接使用神经奖励易引发奖励欺骗。本文提出ReCode(推理增强代码生成)框架,包含两项创新:(1) 对比推理过程奖励学习(CRPL),通过合成优化与退化推理路径训练奖励模型以评估推理质量;(2) 一致性门控GRPO(CG-GRPO),将推理奖励与执行结果强绑定,以执行正确性作为硬门控,防止奖励欺骗。此外,为评估奖励模型对推理过程的判别能力,我们构建了针对代码生成的全新基准LiveCodeBench-RewardBench(LCB-RB),包含成对的优质与劣质推理过程。在HumanEval(+), MBPP(+), LiveCodeBench和BigCodeBench上的实验表明,采用ReCode训练的7B模型相较基线提升16.1%,性能接近GPT-4-Turbo。我们进一步验证了ReCode在数学领域的泛化能力。
原文摘要 · Abstract (English)
In practice, rigorous reasoning is often a key driver of correct code, while Reinforcement Learning (RL) for code generation often neglects optimizing reasoning quality. Bringing process-level supervision into RL is appealing, but it faces two challenges. First, training reliable reward models to assess reasoning quality is bottlenecked by the scarcity of fine-grained preference data. Second, naively incorporating such neural rewards may suffer from reward hacking. This work proposes ReCode (Reasoning-Reinforced Code Generation), a novel RL training framework comprising: (1) Contrastive Reasoning-Process Reward Learning (CRPL), which trains a reward model with synthesized optimized and degraded reasoning variants to assess the quality of reasoning process; and (2) Consistency-Gated GRPO (CG-GRPO), which integrates the reasoning-process reward model into RL by gating neural reasoning-process rewards with strict execution outcomes, using execution correctness as a hard gate to mitigate reward hacking. Additionally, to assess the reward model's discriminative capability in assessing reasoning-process quality, we introduce LiveCodeBench-RewardBench (LCB-RB), a new benchmark comprising preference pairs of superior and inferior reasoning processes tailored for code generation. Experimental results across HumanEval(+), MBPP(+), LiveCodeBench, and BigCodeBench show that a 7B model trained with ReCode outperforms the base version by 16.1% and reaches performance comparable to GPT-4-Turbo. We further demonstrate the generalizability of ReCode by extending it to the math domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。