用强化学习让AI代码自动优化效率，持续超越人类水平。
Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization
- 通过执行反馈闭环，让大模型迭代改进代码。
- 效率提升显著：通过率从47%升至62%，优于人类概率达45%。
- 强化学习比传统方法更有效，实现真正自进化优化。
大型语言模型生成的代码虽功能正确，但在效率上常有不足，制约实际部署。本文提出一种测试时迭代优化框架，构建闭环系统，使大模型基于执行沙箱的性能反馈反复优化代码。对比监督微调（SFT）、直接偏好优化（DPO）与组相对策略优化（GRPO）三种训练策略，实验在Venus数据集和APPS基准上显示，SFT与DPO效率提升迅速饱和；而采用执行反馈的强化学习（GRPO）可持续优化，显著提升通过率（pass@1从47%增至62%），并提高优于人类提交代码的概率（从31%增至45%）。研究证明了测试时代码效率优化的有效性，并揭示强化学习在引导大模型真正自进化代码效率方面的潜力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) generate functionally correct solutions but often fall short in code efficiency, a critical bottleneck for real-world deployment. In this paper, we introduce a novel test-time iterative optimization framework to address this, employing a closed-loop system where LLMs iteratively refine code based on empirical performance feedback from an execution sandbox. We explore three training strategies: Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO). Experiments on our Venus dataset and the APPS benchmark show that SFT and DPO rapidly saturate in efficiency gains. In contrast, GRPO, using reinforcement learning (RL) with execution feedback, continuously optimizes code performance, significantly boosting both pass@1 (from 47% to 62%) and the likelihood of outperforming human submissions in efficiency (from 31% to 45%). Our work demonstrates effective test-time code efficiency improvement and critically reveals the power of RL in teaching LLMs to truly self-improve code efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。