让代码推理可验证:通过逐步执行轨迹训练模型,避免乱猜答案。
StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning

- 在代码中插入打印锚点,监督每步运行状态,使推理过程可追踪。
- 7B模型在CRUXEval上达91.1%,超越GPT-4o和原有基线模型。
- 适合追求可解释性与高准确率的代码生成与推理研究者。
现有代码推理方法主要依赖最终代码输出进行监督,忽略中间状态,常导致奖励劫持——通过不一致推理获得正确答案。我们提出StepCodeReasoner,引入显式中间执行状态监督。通过自动在代码中插入基于print的执行轨迹锚点,模型被训练预测每一步的运行状态,将代码推理转化为可验证的分步执行建模问题。在此基础上,我们设计双层GRPO强化学习算法,实现两级信用分配:跨轨迹比较不同执行路径,及在轨迹内根据中间状态对最终正确性的贡献进行奖励。大量实验表明,StepCodeReasoner达到代码推理最先进水平。特别是,我们的7B模型在CRUXEval上达到91.1%,在LiveCodeBench上达86.5%,优于CodeReasoner-7B(86.0%、77.7%)和GPT-4o(85.6%、75.1%)。在执行轨迹基准REval上,模型得分为82.9%,优于基线CodeReasoner-7B(72.3%)、其14B版本(81.1%)及GPT-4o(77.3%)。此外,该方法也提升代码生成性能,证明显式执行建模同时增强代码推理与生成能力。
原文摘要 · Abstract (English)
Existing code reasoning methods primarily supervise final code outputs, ignoring intermediate states, often leading to reward hacking where correct answers are obtained through inconsistent reasoning. We propose StepCodeReasoner, a framework that introduces explicit intermediate execution-state supervision. By automatically inserting structured print-based execution-trace anchors into code, the model is trained to predict runtime states at each step, transforming code reasoning into a verifiable, stepwise execution modeling problem. Building on this execution-aware method, we introduce Bi-Level GRPO, a reinforcement learning algorithm for structured credit assignment at two levels: inter-trajectory, comparing alternative execution paths, and intra-trajectory, rewarding intermediate accuracy based on its impact on downstream correctness. Extensive experiments demonstrate that StepCodeReasoner achieves SOTA performance in code reasoning. In particular, our 7B model achieves 91.1\% on CRUXEval and 86.5\% on LiveCodeBench, outperforming the CodeReasoner-7B baseline (86.0\% and 77.7\%) and GPT-4o (85.6\% and 75.1\%). Furthermore, on the execution-trace benchmark REval, our model scores 82.9\%, outperforming baseline CodeReasoner-7B (72.3\%), its 14B counterpart (81.1\%), and GPT-4o (77.3\%). Additionally, our approach also improves code generation performance, demonstrating that explicit execution modeling enhances both code reasoning and code generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。