发现SWE-Bench测试用例不足，提出新框架纠正错误评估结果
UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench
- 用LLM自动分析代码生成更全面的测试用例
- 发现345个错误修复被误判为通过，影响超四成基准数据
- 适合关注代码生成评估可信度的研究者与开发者
大型语言模型（LLMs）推动了真实世界代码生成编码代理的发展。作为广泛使用的评估基准，SWE-Bench基于GitHub问题及其对应拉取请求中的实际问题进行评测。然而，这些拉取请求中包含的手动编写测试用例往往不足，导致生成的补丁即使未真正解决问题也能通过测试。为此，我们提出UTGenerator，一种由LLM驱动的测试用例生成器，可自动分析代码库及依赖关系，为真实世界的Python项目生成测试用例。在此基础上，我们构建了UTBoost，一个完整的测试用例增强框架。在评估中，我们识别出36个测试用例不足的任务实例，并发现345个错误补丁在原始SWE-Bench中被错误地标记为通过。这些修正影响了SWE-Bench Lite 40.9%和SWE-Bench Verified 24.4%的排行榜条目，分别导致18次和11次排名变动。
原文摘要 · Abstract (English)
The advent of Large Language Models (LLMs) has spurred the development of coding agents for real-world code generation. As a widely used benchmark for evaluating the code generation capabilities of these agents, SWE-Bench uses real-world problems based on GitHub issues and their corresponding pull requests. However, the manually written test cases included in these pull requests are often insufficient, allowing generated patches to pass the tests without resolving the underlying issue. To address this challenge, we introduce UTGenerator, an LLM-driven test case generator that automatically analyzes codebases and dependencies to generate test cases for real-world Python projects. Building on UTGenerator, we propose UTBoost, a comprehensive framework for test case augmentation. In our evaluation, we identified 36 task instances with insufficient test cases and uncovered 345 erroneous patches incorrectly labeled as passed in the original SWE Bench. These corrections, impacting 40.9% of SWE-Bench Lite and 24.4% of SWE-Bench Verified leaderboard entries, yield 18 and 11 ranking changes, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。