构建大规模自动化代码数据集,发现模型性能随数据量持续提升。
Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs
- 设计自动化流程,从2531个真实仓库提取1万+代码任务。
- 训练模型在基准上达38.0%准确率,数据越多越准无饱和迹象。
- 适合研究大模型代码生成与自动验证的开发者和研究人员。
软件工程(SWE)已成为下一代大模型智能体的关键测试场,需具备持续迭代求解(如超过50轮交互)和长上下文依赖解析(如超过32k token)能力。然而,现有数据集受限于人工标注与运行环境搭建,普遍仅含数千个实例。为此,我们提出一种增量式自动化数据构建流程,系统扩展了数据规模与多样性。所构建数据集包含来自2,531个GitHub仓库的10,169个真实Python任务,每个任务均配有自然语言描述及专用运行环境镜像,支持自动化单元测试验证。从中筛选出超过8,000条成功通过验证的训练轨迹。在这些轨迹上微调Skywork-SWE模型后,发现其性能随数据量增长持续提升,未见饱和。该模型在SWE-bench Verified基准上达到38.0% pass@1准确率,无需验证器或多次采样,成为基于Qwen2.5-Coder-32B与OpenHands框架的SOTA。结合测试时缩放技术,准确率进一步提升至47.0%,超越所有参数量低于32B的模型。我们已公开Skywork-SWE-32B模型检查点以推动后续研究。
原文摘要 · Abstract (English)
Software engineering (SWE) has recently emerged as a crucial testbed for next-generation LLM agents, demanding inherent capabilities in two critical dimensions: sustained iterative problem-solving (e.g., >50 interaction rounds) and long-context dependency resolution (e.g., >32k tokens). However, the data curation process in SWE remains notoriously time-consuming, as it heavily relies on manual annotation for code file filtering and the setup of dedicated runtime environments to execute and validate unit tests. Consequently, most existing datasets are limited to only a few thousand GitHub-sourced instances. To this end, we propose an incremental, automated data-curation pipeline that systematically scales both the volume and diversity of SWE datasets. Our dataset comprises 10,169 real-world Python task instances from 2,531 distinct GitHub repositories, each accompanied by a task specified in natural language and a dedicated runtime-environment image for automated unit-test validation. We have carefully curated over 8,000 successfully runtime-validated training trajectories from our proposed SWE dataset. When fine-tuning the Skywork-SWE model on these trajectories, we uncover a striking data scaling phenomenon: the trained model's performance for software engineering capabilities in LLMs continues to improve as the data size increases, showing no signs of saturation. Notably, our Skywork-SWE model achieves 38.0% pass@1 accuracy on the SWE-bench Verified benchmark without using verifiers or multiple rollouts, establishing a new state-of-the-art (SOTA) among the Qwen2.5-Coder-32B-based LLMs built on the OpenHands agent framework. Furthermore, with the incorporation of test-time scaling techniques, the performance further improves to 47.0% accuracy, surpassing the previous SOTA results for sub-32B parameter models. We release the Skywork-SWE-32B model checkpoint to accelerate future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。