arXiv:2501.07531cs.SEcs.AI2025-01被引 56

谷歌实测智能体修复代码,机器报告的错修复率超七成。

Evaluating Agent-based Program Repair at Google

  • 用类SWE-Agent的智能体在谷歌内部环境自动修代码
  • 对机器报告的漏洞修复成功率73%,人工报告的为25.6%
  • 首次在工业级数据集上建立基准,适合企业研发参考

基于智能体的程序修复通过结合现代大模型的规划、工具使用和代码生成能力,有望端到端自动解决复杂缺陷。已有研究在开源的SWE-Bench数据集上探索了此类方法,如SWE-Agent等。本文评估了在企业场景下应用智能体修复的可行性,构建了一个包含178个来自谷歌问题跟踪系统的漏洞数据集,涵盖78个人工报告和100个机器报告的漏洞。为建立性能基线,我们实现了一个名为Passerine的智能体,其设计思想与SWE-Agent相似,且能适配谷歌开发环境。实验表明,在使用Gemini 1.5 Pro并采样20条轨迹的情况下,Passerine对机器报告的漏洞有73%可生成通过测试的补丁(即合理),对人工报告的漏洞为25.6%。经人工评估后发现,43%的机器报告漏洞和17.9%的人工报告漏洞存在语义等价于真实补丁的修复方案。该结果建立了工业级基准,证明所用数据集在语言多样性、代码规模和修改范围等方面与流行的SWE-Bench存在显著差异。

原文摘要 · Abstract (English)

Agent-based program repair offers to automatically resolve complex bugs end-to-end by combining the planning, tool use, and code generation abilities of modern LLMs. Recent work has explored the use of agent-based repair approaches on the popular open-source SWE-Bench, a collection of bugs from highly-rated GitHub Python projects. In addition, various agentic approaches such as SWE-Agent have been proposed to solve bugs in this benchmark. This paper explores the viability of using an agentic approach to address bugs in an enterprise context. To investigate this, we curate an evaluation set of 178 bugs drawn from Google's issue tracking system. This dataset spans both human-reported (78) and machine-reported bugs (100). To establish a repair performance baseline on this benchmark, we implement Passerine, an agent similar in spirit to SWE-Agent that can work within Google's development environment. We show that with 20 trajectory samples and Gemini 1.5 Pro, Passerine can produce a patch that passes bug tests (i.e., plausible) for 73% of machine-reported and 25.6% of human-reported bugs in our evaluation set. After manual examination, we found that 43% of machine-reported bugs and 17.9% of human-reported bugs have at least one patch that is semantically equivalent to the ground-truth patch. These results establish a baseline on an industrially relevant benchmark, which as we show, contains bugs drawn from a different distribution -- in terms of language diversity, size, and spread of changes, etc. -- compared to those in the popular SWE-Bench dataset.

程序修复智能体工业应用谷歌

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。