评测编程智能体在不改变行为前提下的代码质量优化能力。
SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements
- 构建188个真实开源项目中的代码改进任务,定义92条可执行的非功能优化规则。
- 顶尖智能体功能正确率70.0%,但整体代码优化能力远低于人类开发者。
- 首次系统评估编程智能体在结构化代码改进上的短板,适合研究智能编程工具者参考。
尽管编程智能体在以正确性为导向的基准上表现优异,但其在不改变可观测行为的前提下进行非功能性改进(NFIs)的能力仍缺乏深入探索。在真实软件开发中,开发者持续优化代码质量而不改变功能,而现有基准主要评估功能性正确性,对非功能性改进支持有限。本文提出SWE-NFI,一个用于评估编程智能体在非功能性改进方面能力的基准。该基准包含188个来自开源Python项目真实合并请求的任务,将开发者关注的非功能性改进转化为92条可执行规则,并构建了结合功能正确性测试与规则驱动的NFI评估体系。我们评估了最先进的商用和开源编程智能体。尽管最佳智能体的功能正确率达到70.0%,但所有被评估智能体在整体非功能性改进能力上普遍逊于人类开发者。尤其在结构化代码改进方面,智能体得分范围为0.0至1.3,而人类参考得分则为1.5。本基准与发现为超越功能正确性的编程智能体评估与提升提供了可复现的基础。
原文摘要 · Abstract (English)
Although coding agents have achieved impressive performance on correctness-oriented benchmarks, their ability to make behavior-preserving non-functional improvements (NFIs) remains underexplored. In real-world software development, developers continuously improve software quality without changing observable behavior, yet existing benchmarks primarily evaluate functional correctness and provide limited support for assessing these non-functional improvements. In this paper, we present SWE-NFI, a benchmark for evaluating coding agents on NFIs beyond functional correctness. Our benchmark contains 188 tasks constructed from real merged pull requests in open-source Python projects. We operationalize developer-oriented NFIs into 92 executable rules and develop a comprehensive evaluation suite that combines functional correctness testing with rule-based NFI evaluation. We evaluate state-of-the-art commercial and open-source coding agents. Although the best-performing agent achieves a 70.0\% functional correctness rate, all evaluated agents generally fall short of human developers in overall NFI capability. The gap is particularly evident for structural code improvements, where agents' NFI scores range from 0.0 to 1.3, compared with 1.5 for the human reference. Our benchmark and findings provide a reproducible foundation for evaluating and advancing coding agents beyond functional correctness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。