LLM生成的代码虽能解决复杂问题，但性能波动大且平均不如人类。
Do AI Models Dream of Faster Code? An Empirical Study on LLM-Proposed Performance Improvements in Real-World Software
- 在真实生产代码上测试LLM生成的性能优化方案。
- LLM方案平均性能低于人类开发者，且结果极不稳定。
- 适合关注LLM在工程落地中局限性的研发人员阅读。
大型语言模型（LLMs）能生成代码，但能否为复杂的现实软件系统生成高效代码？本研究基于65个从关键开源Java项目中提取的真实性能敏感任务展开调查。与以往聚焦算法谜题的研究不同，本文在实际生产代码上进行实验，并采用开发者编写的JMH基准测试来严格验证性能提升效果，对比人类基线。结果揭示出复杂现实场景下的真相：尽管LLM展现出惊人解决复杂工程问题的能力，其解决方案仍存在极端不稳定性，整体性能平均落后于人类开发者。因此，当前基于算法任务的基准测试对LLM能力评估过于乐观。我们发现这一差距主要源于两大限制：第一，LLM难以自主定位性能热点；第二，即便获得明确指引，也常无法合成最优算法改进。研究强调，需从静态代码生成转向具备运行时行为分析与观测能力的更复杂智能体系统。
原文摘要 · Abstract (English)
Large Language Models (LLMs) can generate code, but can they generate fast code for complex, real-world software systems? In this study, we investigate this question using a dataset of 65 tasks mined from performance-critical open-source Java projects. Unlike prior studies, which focused on algorithmic puzzles, we conduct experiments on actual performance-sensitive production code and employ developer-written JMH benchmarks to rigorously validate performance gains against human baselines. Our results reveal a nuanced reality -- although LLMs demonstrate a surprisingly high capability to solve these complex engineering problems, their solutions suffer from extreme volatility and still lag behind human developers on average. Consequently, we find that the current benchmarks based on algorithmic tasks yields an overly optimistic assessment of LLM capabilities. We trace this real-world performance gap to two primary limitations: first, LLMs struggle to autonomously pinpoint performance hotspots, and second, even with explicit guidance, they often fall short of synthesizing optimal algorithmic improvements. Our results highlight the need to move beyond static code generation towards more complex agent-based systems that are able to profile and observe runtime behavior for performance improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。