用强化学习让代码大模型学会利用执行反馈迭代优化
RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
- 通过强化学习让大模型在生成代码时主动利用执行反馈
- 8B和70B模型在编程竞赛任务上达新纪录，样本量减少十倍
- 适合需要高效迭代生成代码的开发者与自动化工具研究者
将大语言模型（LLMs）作为智能体解决用户指定任务时，需在多步操作中最小化人工干预。关键在于模型必须基于获得的反馈可靠地调整生成内容。本文提出一种端到端强化学习方法，用于指导模型在代码生成领域有效利用执行反馈，克服当前主流大模型在迭代改进代码方面表现不佳的问题。我们在编程竞赛任务上进行基准测试，结果表明，无论是小模型（8B参数）还是大模型（70B参数），均取得新的最佳性能，同时所需样本数量降低一个数量级。对推理阶段行为的分析显示，该方法使模型能够跨多个步骤有效利用自动反馈。
原文摘要 · Abstract (English)
Large language models (LLMs) deployed as agents solve user-specified tasks over multiple steps while keeping the required manual engagement to a minimum. Crucially, such LLMs need to ground their generations in any feedback obtained to reliably achieve the desired outcomes. We propose an end-to-end reinforcement learning method for teaching models to leverage execution feedback in the realm of code synthesis, where state-of-the-art LLMs struggle to improve code iteratively compared to independent sampling. We benchmark on competitive programming tasks, where we achieve new state-of-the art results with both small (8B parameters) and large (70B) models while reducing the amount of samples required by an order of magnitude. Our analysis of inference-time behavior demonstrates that our method produces LLMs that effectively leverage automatic feedback over multiple steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。