arXiv:2608.09802cs.CLcs.SE2026-08被引 4

新基准测试代码重构,挑战多语言大模型真实工程能力。

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

论文配图:SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
图 1 · 摘自论文原文
  • 构建多语言代码重构任务,每项平均改11.4个文件、261.6行代码。
  • 通过人工审核修复测试用例缺陷,解决旧基准中60%测试不靠谱问题。
  • 适合评估前沿编码代理在复杂跨文件重构中的真实表现。

随着AI编程代理承担更复杂的长周期软件工程任务,现有基准已迅速饱和且评估质量受到质疑:最近审计发现近60%未解决的SWE-bench Verified实例存在缺陷测试——或过窄排斥正确解,或过宽包含未明示需求。此外,前沿模型可直接复现训练数据中的黄金补丁。代码重构要求跨多个文件协调、保持行为一致地修改代码,是更难且更真实的代理能力测试,但当前基准仍严重不足。我们提出SWE-Bench ProMax,一个由专家精心策划的多语言代码重构基准,涵盖来自七种语言(Python、Java、TypeScript、Go、C、C++、Rust)真实提交的170个实例。每个实例均经过多阶段严格筛选:问题描述重新编写以确保精准明确,测试套件人工审查剔除过窄或过宽测试;低复杂度或局限跨文件范围的任务被过滤,最终形成具有挑战性、大规模的重构任务,平均每个实例涉及11.4个文件和261.6行代码,显著超越现有基准规模。在两种代理框架下对前沿模型的实验表明,最佳模型仅达41.2%解决率,证实SWE-Bench ProMax为当前AI编程代理提供了有意义且未饱和的挑战。该基准可在https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax获取。

原文摘要 · Abstract (English)

As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements -- and that frontier models can verbatim reproduce gold patches from training data. Code refactoring, which requires coordinated, behavior-preserving changes across many files, offers a substantially harder and more realistic test of agent capability, yet remains underserved by current benchmarks. We introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark of 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust). Every instance undergoes rigorous, multi-stage curation that directly addresses the quality problems identified in prior benchmarks: issue descriptions are rewritten from scratch to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out, yielding a benchmark of challenging, large-scale refactoring tasks that average 11.4 modified files and 261.6 lines of code per instance, substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents. Our benchmark is available at https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax.

代码重构多语言基准测试AI编程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。