通过执行轨迹对齐提升代码生成的正确性,解决大模型语义偏差问题。
CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment
- 引入变量级执行轨迹作为强化学习信号,增强文本与执行语义的对齐
- 在pass@1上提升4.6%,代码推理和测试生成任务分别提高15.5%和4.4%
- 兼容多种强化学习算法与大模型,适合注重代码功能正确的研究者
尽管大型语言模型(LLMs)通过海量代码语料训练在代码生成上表现优异,但其基于文本模式的训练与功能性正确性之间仍存在根本性语义鸿沟,后者由正式执行语义决定。现有可验证奖励强化学习(RLVR)方法依赖测试用例的通过/失败二值反馈,难以有效建立代码文本表示与执行语义之间的精准关联,尤其对细微逻辑错误效果有限。本文提出CodeRL+,将执行语义对齐融入RLVR训练流程。该方法使模型能够推断变量级别的执行轨迹,提供直接的执行语义学习信号。CodeRL+可利用现有在线策略采样构建语义对齐,并与多种强化学习算法无缝集成。大量实验表明,CodeRL+优于后训练基线(包括RLVR和蒸馏),在pass@1上实现4.6%的平均相对提升;在代码推理与测试输出生成任务上分别取得15.5%和4.4%的准确率增长。该方法在不同强化学习算法与大模型间具有强泛化能力。探针分析进一步证明,CodeRL+显著增强了代码文本表示与其底层执行语义之间的对齐度。
原文摘要 · Abstract (English)
While Large Language Models (LLMs) excel at code generation by learning from vast code corpora, a fundamental semantic gap remains between their training on textual patterns and the goal of functional correctness, which is governed by formal execution semantics. Reinforcement Learning with Verifiable Rewards (RLVR) approaches attempt to bridge this gap using outcome rewards from executing test cases. However, solely relying on binary pass/fail signals is inefficient for establishing a well-aligned connection between the textual representation of code and its execution semantics, especially for subtle logical errors within the code. In this paper, we propose CodeRL+, a novel approach that integrates execution semantics alignment into the RLVR training pipeline for code generation. CodeRL+ enables the model to infer variable-level execution trajectory, providing a direct learning signal of execution semantics. CodeRL+ can construct execution semantics alignment directly using existing on-policy rollouts and integrates seamlessly with various RL algorithms. Extensive experiments demonstrate that CodeRL+ outperforms post-training baselines (including RLVR and Distillation), achieving a 4.6% average relative improvement in pass@1. CodeRL+ generalizes effectively to other coding tasks, yielding 15.5% and 4.4% higher accuracy on code-reasoning and test-output-generation benchmarks, respectively. CodeRL+ shows strong applicability across diverse RL algorithms and LLMs. Furthermore, probe analyses provide compelling evidence that CodeRL+ strengthens the alignment between code's textual representations and its underlying execution semantics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。