用单步奖励实现多轮代码生成，简单高效。
Multi-Turn Code Generation Through Single-Step Rewards
- 将多轮代码生成视为可单步恢复的MDP，仅用单步奖励训练。
- 在多轮反馈下生成代码，性能超越现有最先进方法。
- 适合需要高效利用执行反馈的自动化编程场景。
我们解决从多轮执行反馈中生成代码的问题。现有方法要么不使用反馈，要么采用复杂的分层强化学习来优化多轮奖励。我们提出一种简单且可扩展的方法μCode，仅通过单步奖励即可解决多轮代码生成问题。核心洞察是：代码生成是一个单步可恢复的马尔可夫决策过程，任何中间代码状态都能在单轮内恢复为正确代码。μCode迭代训练一个生成器（基于多轮执行反馈生成代码）和一个验证器（对新生成代码打分）。实验评估表明，该方法显著优于当前最先进的基线模型。我们分析了奖励模型与策略的设计选择，并验证了μCode有效利用执行反馈的能力。代码已开源：https://github.com/portal-cornell/muCode。
原文摘要 · Abstract (English)
We address the problem of code generation from multi-turn execution feedback. Existing methods either generate code without feedback or use complex, hierarchical reinforcement learning to optimize multi-turn rewards. We propose a simple yet scalable approach, $μ$Code, that solves multi-turn code generation using only single-step rewards. Our key insight is that code generation is a one-step recoverable MDP, where the correct code can be recovered from any intermediate code state in a single turn. $μ$Code iteratively trains both a generator to provide code solutions conditioned on multi-turn execution feedback and a verifier to score the newly generated code. Experimental evaluations show that our approach achieves significant improvements over the state-of-the-art baselines. We provide analysis of the design choices of the reward models and policy, and show the efficacy of $μ$Code at utilizing the execution feedback. Our code is available at https://github.com/portal-cornell/muCode.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。