构建跨语言的可复现软件工程任务数据集,支持大规模训练智能编程助手。
SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
- 通过自动化流水线采集真实项目任务,生成可执行环境。
- 建成32,079个跨20语言的任务数据集,12万+任务含安装与测试信息。
- 适合训练多语言编程代理,尤其关注跨语言泛化能力的研究者。
软件工程智能体(SWE)进展迅速,主要由强化学习(RL)驱动。然而,受限于缺乏大规模、可复现且测试可靠的任务集合,RL训练面临瓶颈。尽管已有多个基准,但可用于训练的数据集在规模、多样性上仍不足,或仅针对高资源语言生态。本文提出SWE-rebench V2,一个面向多语言的自动化任务采集与环境构建管道。该管道通过交互式设置代理生成仓库特异性的安装与测试流程,并利用多个LLM判官过滤无效实例,验证基于人工标注的SWE-bench数据。据此构建了包含32,079个任务的数据集,覆盖20种语言和3,617个仓库,配有预构建执行镜像。为进一步扩充训练数据,额外发布12万余个任务,含安装说明、失败到通过的测试及丰富元数据,问题描述源自原始PR内容。通过诊断研究验证数据质量,涵盖5种语言下7个主流模型,提供实例级元数据标记常见混淆因素如测试过严、需求不明确。所有数据集、采集与执行代码及相关资产均已公开,以支持跨语言、跨仓库的大规模SWE智能体训练。
原文摘要 · Abstract (English)
Software engineering agents (SWE) are improving rapidly, with recent gains largely driven by reinforcement learning (RL). However, RL training is constrained by the scarcity of large-scale task collections with reproducible execution environments and reliable test suites. Although a growing number of benchmarks have emerged, datasets suitable for training remain limited in scale and diversity or often target a limited set of high-resource language ecosystems. We introduce SWE-rebench V2, a language-agnostic automated pipeline for harvesting executable real-world SWE tasks and constructing RL training environments at scale. The pipeline synthesizes repository-specific installation and test procedures via an interactive setup agent, and filters unsound instances using an ensemble of LLM judges, validated against human-verified SWE-bench annotations. Using this pipeline, we construct a dataset of 32,079 tasks spanning 20 languages and 3,617 repositories, with pre-built images for reproducible execution. To further scale training data, we additionally release 120,000+ tasks with installation instructions, fail-to-pass tests and rich metadata, where the problem statement is generated based on the original pull request description. We validate the collected instances through a diagnostic study that covers a subset of tasks in five programming languages across seven popular models, and provide instance-level metadata that flags common confounders such as overly restrictive tests and underspecified descriptions. We release the datasets, the collection and execution code, and associated artifacts to enable large-scale training of SWE agents across diverse languages and repositories.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。