用细粒度反馈提升代码生成的强化学习方法
Process Supervision-Guided Policy Optimization for Code Generation
- 在代码生成过程中提供逐行正确性反馈，替代传统延迟奖励
- 实验显示该方法在长序列任务中显著提升生成成功率
- 适合需要精准调试与迭代优化的编程自动化场景
基于单元测试反馈的强化学习已提升大语言模型的代码生成能力，但依赖于完整代码评估后才提供的稀疏奖励，限制了学习效率和增量改进。当生成代码全部测试失败时，无法获得学习信号，阻碍复杂任务进展。为此，我们提出过程奖励模型（Process Reward Model, PRM），在代码生成过程中提供细粒度、行级的正确性反馈，模拟人类代码修正行为并提供即时指导。我们探索了PRM的多种训练策略及其在强化学习框架中的集成方式，发现将PRM同时用作密集奖励和价值函数初始化能显著提升性能。实验结果表明，PRM在增强强化学习驱动的代码生成方面效果显著，尤其适用于长序列生成场景。
原文摘要 · Abstract (English)
Reinforcement learning (RL) with unit test feedback has enhanced large language models' (LLMs) code generation, but relies on sparse rewards provided only after complete code evaluation, limiting learning efficiency and incremental improvements. When generated code fails all unit tests, no learning signal is received, hindering progress on complex tasks. To address this, we propose a Process Reward Model (PRM) that delivers dense, line-level feedback on code correctness during generation, mimicking human code refinement and providing immediate guidance. We explore various strategies for training PRMs and integrating them into the RL framework, finding that using PRMs both as dense rewards and for value function initialization significantly boosts performance. Our experimental results also highlight the effectiveness of PRMs in enhancing RL-driven code generation, especially for long-horizon scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。