用编译执行结果自动标注代码生成过程,提升模型质量
Process-Supervised Reinforcement Learning for Code Generation
- 通过教师模型逐行改写代码,用编译执行结果自动标注每行
- 在多个基准上,过程监督比仅结果监督显著更优
- 适合需要严谨代码流程与正确性的复杂生成任务
基于结果监督的强化学习已在代码生成中证明有效。然而,基于过程监督的强化学习在代码生成中的效果仍待深入探索。主要障碍在于高质量过程监督数据构建成本高,需大量人工与算力。为此,我们提出“语句变异/重构-编译与执行验证”策略:通过教师模型逐行改写代码,并利用编译执行结果自动标注每行,生成逐行过程监督数据,用于训练过程监督奖励模型。该模型被集成至PRLCoder框架,在多个基准上进行实验验证。结果表明,过程监督强化学习显著优于仅依赖结果监督的方法。尤其在处理复杂代码生成任务时,其优势明显,既能保障生成过程完整性,又能确保结果正确性。
原文摘要 · Abstract (English)
Existing reinforcement learning strategies based on outcome supervision have proven effective in enhancing the performance of large language models(LLMs) for code generation. While reinforcement learning based on process supervision has shown great promise in handling multi-step reasoning tasks, its effectiveness in code generation remains largely underexplored and underjustified. The primary obstacle stems from the resource-intensive nature of constructing high-quality process-supervised data, which demands substantial human expertise and computational resources. In response to this challenge, we propose a "statement mutation/refactoring-compile and execution verification" strategy: mutating and refactoring code line-by-line through a teacher model, and utilizing compiler execution results to automatically label each line, resulting in line-by-line process-supervised data, which is pivotal for training a process-supervised reward model. The trained reward model is then integrated into the PRLCoder framework, followed by experimental validation on several benchmarks. Experimental results demonstrate that process-supervised reinforcement learning significantly surpasses methods relying solely on outcome supervision. Notably, in tackling complex code generation tasks, process-supervised reinforcement learning shows a clear advantage, ensuring both the integrity of the code generation process and the correctness of the generation results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。