三大代码优化基准测试存在可靠性问题，实测发现多数结果受机器环境和评分规则干扰。
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
- 重测4种云服务器上740个任务，仅约四成基准任务的参考补丁在不同环境下仍有效
- 同一提交在不同评分规则下排名差异显著，某基准对最差任务赋予超80%权重
- 超八成任务已有公开提交超越基准表现，但聚合排名掩盖了真实性能差距
GSO、SWE-Perf 和 SWE-fficiency 等仓库级性能优化基准通过向真实仓库应用补丁并比较运行时长来评估代码智能体。这些基准的排行榜分数正被广泛用作代码智能体进展的证据，但分数可能混杂了运行时不稳定、特定评分规则以及已有多个公开提交解决任务等因素。我们对这三个基准进行了审计：首先，在四种常见Google Cloud机器上重播740个代码优化任务的官方参考补丁，发现只有39/102个GSO任务、11/140个SWE-Perf任务、411/498个SWE-fficiency任务在跨机器重播中仍满足原始有效性标准；其中SWE-Perf尤其脆弱，许多参考补丁导致近零运行时变化。其次，我们发现公开提交的排名高度依赖评分规则，在8个同时参与GSO和SWE-fficiency的提交中，官方排名在28组对比中有9组不一致，且SWE-fficiency的评分规则为最差的10个任务分配了58.5%–82.8%的过高权重。第三，分析每项任务的10个公开提交，发现至少有一个提交在384/450个重播有效的任务中达到或超过参考补丁表现，且在449/450个任务中优于未优化基线代码。本研究补充了排行榜分数，识别出更具可靠性能信号的任务，量化各任务得分贡献，并揭示被聚合排名掩盖的真实性能差距。
原文摘要 · Abstract (English)
Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these issues across the three benchmarks. First, we replay the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every cross-machine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime changes. Second, we show that public submission rankings depend strongly on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency's leaderboard scoring rule assigns the worst ten tasks overly high score weights of 58.5%-82.8%. Third, looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450). Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing the remaining performance gaps that are hidden by aggregate rankings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。