评测大模型优化真实代码库的能力，发现其性能提升远低于专家水平。
SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?
- 构建真实代码库的性能优化任务集，要求定位瓶颈并修复代码。
- 平均仅达专家速度提升的0.23倍，暴露出模型在推理与正确性维护上的短板。
- 适合研究自动化软件工程、长程代码理解与性能优化的学者使用。
大规模软件仓库的性能优化需要代码推理与软件工程知识，以减少运行时间同时保持程序正确性。然而，现有基准多关注‘改什么’而非‘怎么改’。我们提出SWE-fficiency，一个在真实工作负载下评估仓库级性能优化的基准。该套件涵盖9个广泛使用的数据科学、机器学习与高性能计算仓库（如numpy、pandas、scipy）中的498个任务：给定完整代码库和慢速工作负载，智能体需分析代码语义、定位性能瓶颈及关联测试，生成能匹配或超越专家提速效果且通过相同单元测试的补丁。为实现‘如何修复’的评估，我们设计自动化流水线，从GitHub拉取请求中提取性能改进操作，结合关键词过滤、静态分析、覆盖率工具与执行验证，确认专家提速基线并识别相关单元测试。对前沿智能体的实证评估显示显著不足：平均仅达专家提速的0.23倍，主要问题在于瓶颈定位、跨函数执行推理以及修改后正确性的维持。我们公开该基准与配套数据流水线，推动自动化性能工程与长周期软件推理研究。
原文摘要 · Abstract (English)
Optimizing the performance of large-scale software repositories demands expertise in code reasoning and software engineering (SWE) to reduce runtime while preserving program correctness. However, most benchmarks emphasize what to fix rather than how to fix code. We introduce SWE-fficiency, a benchmark for evaluating repository-level performance optimization on real workloads. Our suite contains 498 tasks across nine widely used data-science, machine-learning, and HPC repositories (e.g., numpy, pandas, scipy): given a complete codebase and a slow workload, an agent must investigate code semantics, localize bottlenecks and relevant tests, and produce a patch that matches or exceeds expert speedup while passing the same unit tests. To enable this how-to-fix evaluation, our automated pipeline scrapes GitHub pull requests for performance-improving edits, combining keyword filtering, static analysis, coverage tooling, and execution validation to both confirm expert speedup baselines and identify relevant repository unit tests. Empirical evaluation of state-of-the-art agents reveals significant underperformance. On average, agents achieve less than 0.23x the expert speedup: agents struggle in localizing optimization opportunities, reasoning about execution across functions, and maintaining correctness in proposed edits. We release the benchmark and accompanying data pipeline to facilitate research on automated performance engineering and long-horizon software reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。