用可执行验证统一过程与结果奖励，提升代码生成的准确性和效率
Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation
- 构建树状搜索框架，通过执行指标和自省评分筛选候选代码
- 在3个基准上实现26.9%正确率提升、42.2%代码效率改进
- 适合需要复杂推理的编程任务，尤其对强化学习训练者有参考价值
大型语言模型在代码生成方面表现优异，但在需要复杂推理的编程任务中仍存在瓶颈。传统过程监督依赖需大量标注数据的奖励模型，易出现奖励错位；而仅靠结果监督又难以应对需协调中间步骤的复杂任务。为此，本文提出成果优化过程监督（ORPS），通过可执行验证统一过程与结果监督：采用树状搜索框架生成策略性备选方案，分析执行指标，并通过融合运行时反馈与推理的自省机制对候选代码进行评分。在5个模型和3个基准上的实验表明，该方法带来一致性能提升，正确率提高26.9%，代码效率提升42.2%。结果证明，ORPS能帮助大模型克服代码生成中的局部最优问题，为结合可验证结果与结构化推理解决复杂挑战提供了新方向。代码已开源：https://github.com/zhuohaoyu/ORPS
原文摘要 · Abstract (English)
Large Language Models excel at code generation yet struggle with complex programming tasks that demand sophisticated reasoning. To bridge this gap, traditional process supervision relies on learned reward models requiring costly training data and suffering from reward misalignment, while outcome supervision fails for complex tasks needing coordinated intermediate steps. We introduce Outcome Refining Process Supervision, which unifies process and outcome supervision by leveraging executable verification: a tree-structured search framework generates strategic alternatives, profiles execution metrics, and scores candidates via self-critique mechanisms that integrate runtime feedback with reasoning. Experiments across 5 models and 3 benchmarks show consistent gains, with 26.9% higher correctness and 42.2% improved code efficiency. The results demonstrate that ORPS enables LLMs to overcome local optima in code generation, suggesting a promising direction for combining verifiable outcomes with structured reasoning to tackle complex challenges. We open-source at: https://github.com/zhuohaoyu/ORPS
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。