系统梳理273个软件工程AI基准，提出搜索工具与统一框架提升评估质量。
Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Quality
- 构建语义搜索工具BenchScout，通过上下文嵌入自动聚类定位适配基准。
- 提出BenchFrame框架，使HumanEval基准测试难度提升，模型性能平均下降31.22%。
- 适用于关注基准可靠性、模型评估严谨性的研究者与开发者。
基准测试对统一评估和可复现性至关重要。人工智能赋能软件工程（AI4SE）的快速发展催生了大量代码生成、缺陷修复等任务的基准。然而，其泛滥导致四大挑战：任务间知识碎片化、难以选择适配基准、创建标准缺失、存在缺陷影响实用性。为此，我们提出双重方案：系统梳理现有基准以辅助选型，并制定统一指南以提升基准开发质量。我们分析了247项研究，识别出自2014年以来的273个AI4SE基准，进行分类、分析局限并揭示实践空白。基于此，我们开发了可扩展的语义搜索工具BenchScout，采用基准相关研究的上下文嵌入进行自动聚类，再经降维处理。22名用户的使用研究显示，BenchScout在可用性、有效性、直观性上分别获4.5、4.0、4.1分（满分5分）。为提升基准标准，我们提出BenchFrame统一框架。将其应用于HumanEval，生成HumanEvalNext，具备修正错误、改进语言转换、更高测试覆盖率及更大难度。在10个先进代码模型上评估，相比原版HumanEval，pass-at-1平均下降31.22%；在HumanEvalPlus上下降19.94%，凸显持续优化基准的必要性。我们还通过代理流水线验证了BenchFrame的可扩展性，并在MBPP数据集上确认其通用性。所有评审数据、用户研究材料与增强基准均已公开。
原文摘要 · Abstract (English)
Benchmarks are essential for unified evaluation and reproducibility. The rapid rise of Artificial Intelligence for Software Engineering (AI4SE) has produced numerous benchmarks for tasks such as code generation and bug repair. However, this proliferation has led to major challenges: (1) fragmented knowledge across tasks, (2) difficulty in selecting contextually relevant benchmarks, (3) lack of standardization in benchmark creation, and (4) flaws that limit utility. Addressing these requires a dual approach: systematically mapping existing benchmarks for informed selection and defining unified guidelines for robust, adaptable benchmark development. We conduct a review of 247 studies, identifying 273 AI4SE benchmarks since 2014. We categorize them, analyze limitations, and expose gaps in current practices. Building on these insights, we introduce BenchScout, an extensible semantic search tool for locating suitable benchmarks. BenchScout employs automated clustering with contextual embeddings of benchmark-related studies, followed by dimensionality reduction. In a user study with 22 participants, BenchScout achieved usability, effectiveness, and intuitiveness scores of 4.5, 4.0, and 4.1 out of 5. To improve benchmarking standards, we propose BenchFrame, a unified framework for enhancing benchmark quality. Applying BenchFrame to HumanEval yielded HumanEvalNext, featuring corrected errors, improved language conversion, higher test coverage, and greater difficulty. Evaluating 10 state-of-the-art code models on HumanEval, HumanEvalPlus, and HumanEvalNext revealed average pass-at-1 drops of 31.22% and 19.94%, respectively, underscoring the need for continuous benchmark refinement. We further examine BenchFrame's scalability through an agentic pipeline and confirm its generalizability on the MBPP dataset. All review data, user study materials, and enhanced benchmarks are publicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。