用运行反馈提升大模型生成代码的效率，让代码跑得更快。
PerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedback
- 在测试执行中获取运行时反馈，用于迭代优化代码性能。
- 在多个基准上超越基线模型，部分任务速度超过参考解。
- 适用于各类开源大模型，无需训练即可提升效率。
大型语言模型（LLMs）广泛用于辅助软件开发，但现有评估多关注生成代码的功能正确性。而实际编程中，程序员不仅需要代码正确，还需高效运行。我们提出 PerfCodeGen，一种无需训练的框架，通过在测试用例执行过程中引入运行时反馈，驱动自修正迭代以提升代码性能。相比使用复杂提示技巧的基线 LLM，PerfCodeGen 在更多问题上实现了显著加速。应用于 Phi-3-mini 等开源模型时，其运行效率接近 GPT-4 等闭源强模型表现。在 HumanEval、MBPP、APPS 等基准上达到当前最优的运行效率，常优于真实参考解。该方法对不同规模的开源模型（包括 Phi-3-mini、Llama 3 8B、Mixtral 8x7B、Command R、Llama 3 70B）均有效。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are widely adopted for assisting in software development tasks, yet their performance evaluations have narrowly focused on the functional correctness of generated code. Human programmers, however, require LLM-generated code to be not only correct but also optimally efficient. We propose PerfCodeGen, a training-free framework that enhances the performance of LLM-generated code by incorporating feedback based on runtime during test case execution into the self-refinement iterations. With PerfCodeGen, we achieve speedups for a significantly higher proportion of problems compared to using the base LLM with sophisticated prompting techniques. Applied to open language models like Phi-3-mini, PerfCodeGen achieves runtime efficiency comparable to prompting powerful closed models like GPT-4. We achieve state-of-the-art runtime efficiency on benchmarks such as HumanEval, MBPP, and APPS, frequently surpassing the ground truth reference solutions with PerfCodeGen using GPT-3.5 and GPT-4. Additionally, we demonstrate the effectiveness of our approach in enhancing code quality across a range of open LLMs of varying sizes including Phi-3-mini, Llama 3 8B, Mixtral 8x7B, Command R, and Llama 3 70B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。