评测编程智能体优化高性能代码的能力，发现顶尖模型成功率不足5%。
GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents
- 构建自动化流水线，从真实代码库中提取102个高难度优化任务。
- 顶尖智能体在性能提升上成功率低于5%，即使扩大推理规模也改善有限。
- 适合研究编程智能体、软件优化与自动调优的学者与开发者。
开发高性能软件是一项复杂任务，需要专业知识。我们提出GSO基准，用于评估语言模型在编写高性能代码方面的能力。通过自动化流水线生成并执行性能测试，分析仓库提交历史，从10个代码库中识别出跨多个领域和编程语言的102个挑战性优化任务。智能体接收代码库和性能测试作为精确规范，目标是提升运行效率，以专家开发者优化结果为参照。定量评估显示，领先SWE-Agents表现显著不足，成功率低于5%，即使在推理时扩展也仅带来有限改进。定性分析揭示主要失败模式：难以处理低级语言、采用懒惰优化策略，以及无法准确定位性能瓶颈。我们公开了基准代码、数据及智能体轨迹，以支持未来研究。
原文摘要 · Abstract (English)
Developing high-performance software is a complex task that requires specialized expertise. We introduce GSO, a benchmark for evaluating language models' capabilities in developing high-performance software. We develop an automated pipeline that generates and executes performance tests to analyze repository commit histories to identify 102 challenging optimization tasks across 10 codebases, spanning diverse domains and programming languages. An agent is provided with a codebase and performance test as a precise specification, and tasked to improve the runtime efficiency, which is measured against the expert developer optimization. Our quantitative evaluation reveals that leading SWE-Agents struggle significantly, achieving less than 5% success rate, with limited improvements even with inference-time scaling. Our qualitative analysis identifies key failure modes, including difficulties with low-level languages, practicing lazy optimization strategies, and challenges in accurately localizing bottlenecks. We release the code and artifacts of our benchmark along with agent trajectories to enable future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。