修复代码代理评测基准的漏洞，让评估更真实可靠。
SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents
- 通过防作弊机制消除答案泄露，防止模型投机取巧。
- 修正任务描述和测试范围问题，提升评测质量。
- 发现部分模型性能远低于原报告，适合严谨评估者使用。
SWE-Bench Pro已成为评估软件工程代理在复杂仓库级任务上表现的标准基准。然而，我们的分析表明其评估受两大不可靠因素影响：一是奖励黑客行为，源于正确答案或隐藏评估信息的泄露；二是任务质量缺陷，包括误导性问题陈述和不当的测试范围。这些问题会虚高模型表现，掩盖其真实编码能力。本文提出SWE-Bench Pro Verified，通过反作弊防护机制消除主要信息泄露渠道，同时对存在缺陷的任务实例进行最小化修正。评估结果显示，部分模型在新基准上的表现显著下降，暗示原有SWE-Bench Pro结果可能高估了真实能力。SWE-Bench Pro Verified为评估软件工程代理提供了更可信的基准。
原文摘要 · Abstract (English)
SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluation is undermined by two sources of unreliability: \textbf{reward hacking}, enabled by leakage of gold solutions or hidden evaluation information, and \textbf{task quality issues}, including misleading problem statements and improperly scoped tests. These issues can inflate benchmark performance and obscure agents' true coding ability. We present \textbf{SWE-Bench Pro Verified}, a verified version of SWE-Bench Pro that addresses both problems. Our approach combines \textbf{anti-hacking} safeguards that eliminate major leakage channels without disrupting normal agent functionality, with \textbf{task refinement} that minimally corrects inconsistencies within flawed instances. Evaluations on SWE-Bench Pro Verified reveal that some models perform substantially worse than previously reported, suggesting that existing results on SWE-Bench Pro may overestimate real software engineering capability. SWE-Bench Pro Verified offers a more trustworthy benchmark for assessing software engineering agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。