首个面向真实软件功能开发的大型数据集，助力自主编程模型训练与评估。
SWE-Dev: Evaluating and Training Autonomous Feature-Driven Software Development
- 构建14000样本数据集，含可运行环境与开发者编写的单元测试。
- 单轮模型在难题上仅达22.51%通过率，多智能体系统提升至56.44%。
- 适合研究自主编程、强化学习编码及工具增强大模型的学者使用。
大语言模型在多样化的软件工程任务中展现出强大能力，但面向大型现有代码库的功能开发这一普遍实践仍缺乏研究。为此，我们提出SWE-Dev，首个大规模数据集（包含14,000个训练样本和500个测试样本），用于评估与训练自主编程系统在真实世界端到端功能开发任务上的表现。为确保可验证性和多样性，SWE-Dev为每个实例提供可运行环境及开发者编写的可执行单元测试。该数据集不仅适用于监督微调（SFT），还可通过可执行单元测试提供精准奖励信号，支持强化学习（RL）。我们在17个基础模型、10个推理优化模型、10个多智能体系统及8个工具增强大模型代理上评估了SWE-Dev。结果表明仍有巨大提升空间：最佳单轮模型在困难划分上仅达22.51% Pass@1，而OpenHands代理提升至56.44%，但仍无法完成多数任务。代码已开源：https://github.com/DorothyDUUU/SWE-Dev。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown strong capability in diverse software engineering tasks. However, feature-driven development, a highly prevalent real-world task that involves developing new functionalities for large, existing codebases, remains underexplored. We therefore introduce SWE-Dev, the first large-scale dataset (with 14,000 training and 500 test samples) designed to evaluate and train autonomous coding systems on real-world end-to-end feature-driven software development tasks. To ensure verifiable and diverse training, SWE-Dev uniquely provides all instances with a runnable environment and its developer-authored executable unit tests. This collection not only provides high-quality data for Supervised Fine-Tuning (SFT), but also enables Reinforcement Learning (RL) by delivering accurate reward signals from executable unit tests. We evaluated SWE-Dev across 17 base LLMs, 10 reasoning-focused LLMs, 10 multi-agent systems, and 8 tool-augmented LLM agents. Results show substantial headroom: the best single-turn model reaches only 22.51\% Pass@1 on the hard split, while OpenHands agents improve to 56.44\% but still leave many tasks unsolved. Code is available here https://github.com/DorothyDUUU/SWE-Dev.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。