用真实开发习惯生成复杂漏洞，训练效率提升一倍
BugPilot: Complex Bug Generation for Efficient Learning of SWE Skills
- 让SWE智能体主动引入功能导致测试失败，模拟真实漏洞产生
- 仅用1.2k个漏洞，训练效果比3k个传统漏洞高2%准确率
- 适合想高效训练代码智能体的研究者和开发者
高质量漏洞是训练基于语言模型的软件工程（SWE）智能体的关键。我们提出一种新型合成方法，用于生成复杂且多样化的漏洞。该方法指导SWE智能体向代码库中引入新功能，从而无意间破坏测试，产生漏洞。以往方法常通过局部扰动现有代码刻意制造漏洞，造成分布外问题，不符合真实开发流程。定性分析显示，我们的生成方式更贴近人工修改模式。大量实验表明，使用本方法生成的漏洞可实现更高效的监督微调，在仅用1.2k个漏洞（对比其他数据集3k个）的情况下，性能领先2%。我们在新生成漏洞基础上结合已有数据集训练出FrogBoss（32B参数）与FrogMini（14B参数），分别在SWE-bench Verified上达到54.6%和45.3%的pass@1得分（三次种子平均），达到当前最优水平。
原文摘要 · Abstract (English)
High quality bugs are key to training the next generation of language model based software engineering (SWE) agents. We introduce a novel method for synthetic generation of difficult and diverse bugs. Our method instructs SWE Agents to introduce a feature into the codebase whereby they may unintentionally break tests, resulting in bugs. Prior approaches often induce an out-of-distribution effect by generating bugs intentionally (e.g. by introducing local perturbation to existing code), which does not reflect realistic development processes. We perform qualitative analysis to demonstrate that our approach for generating bugs more closely reflects the patterns found in human-authored edits. Through extensive experiments, we demonstrate that our bugs provide more efficient training data for supervised fine-tuning, outperforming other bug datasets by 2% with half the training data (1.2k vs. 3k bugs). We train on our newly generated bugs in addition to existing bug datasets to get FrogBoss a state-of-the-art 32B parameter model on SWE-bench Verified with a pass@1 of 54.6% and FrogMini a state-of-the-art 14B model on SWE-bench Verified with a pass@1 of 45.3% on SWE-bench Verified averaged over three seeds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。