自动化生成多语言代码任务基准，覆盖真实开源项目中的缺陷修复与功能新增。
SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories
- 从真实 GitHub PR 提取任务，自动构建可复现的编码评估场景。
- 涵盖11种语言共11,133个任务，最强模型在子集上最高通过率仅36.2%。
- 支持微调提升模型性能，适合研究多语言代码生成与评测的团队。
SWE-bench 等基准已标准化大语言模型在仓库级软件工程任务上的评估。然而，这些工作受限于人工标注、静态数据集以及对 Python 编辑修复的偏倚。我们提出 SWE-Bench++，一个从开源 GitHub 项目自动构建仓库级编码任务的框架。不同于合成方法，该流水线提取实时的拉取请求（PR），覆盖 11 种语言的缺陷修复与功能请求。SWE-Bench++ 通过四个阶段将 GitHub PR 转化为可复现的执行型任务：程序化获取、环境合成、测试断言提取和质量保障。最后一步提示引导轨迹合成，将强模型失败的任务转化为训练轨迹。初始基准包含来自 3,971 个仓库的 11,133 个实例。在其中 1,782 个实例的子集上，当前最强模型表现如下：claude-sonnet-4.5 达到 36.20% pass@10，gpt-5-2025-08-07 为 34.57%，gemini/gemini-2.5-pro 为 24.92%，gpt-4o 为 16.89%。我们进一步验证了其有效性：在 SWE-Bench++ 实例上微调模型，能显著提升 SWE-bench Multilingual 基准的表现。SWE-Bench++ 为评估和改进仓库级代码生成提供了可扩展、多语言的基准。
原文摘要 · Abstract (English)
Benchmarks like SWE-bench have standardized the evaluation of Large Language Models (LLMs) on repository-level software engineering tasks. However, these efforts remain limited by manual curation, static datasets, and a focus on Python-based bug fixes. We introduce SWE-Bench++, an automated framework that generates repository-level coding tasks from open-source GitHub projects. Unlike synthetic approaches, our pipeline harvests live pull requests to cover both bug fixes and feature requests across 11 languages. SWE-Bench++ turns GitHub pull requests (PRs) into reproducible, execution-based tasks via four stages: programmatic sourcing, environment synthesis, test oracle extraction, and quality assurance. A final hint-guided trajectory synthesis step converts instances that strong models fail on into training trajectories. Our initial benchmark consists of 11,133 instances from 3,971 repositories across 11 languages. On a subset of 1,782 instances of this benchmark, today's strongest models perform as follows: claude-sonnet-4.5 achieves 36.20% pass@10, gpt-5-2025-08-07 34.57%, gemini/gemini-2.5-pro 24.92%, and gpt-4o 16.89%. We further demonstrate the utility of our dataset by showing that fine-tuning on SWE-Bench++ instances yields measurable improvements on the SWE-bench Multilingual benchmark. SWE-Bench++ provides a scalable, multilingual benchmark for evaluating and improving repository-level code generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。