对比6个基于大模型的智能体在自动修复漏洞上的表现，发现仍需优化模型能力和任务流程。
An Empirical Study on LLM-based Agents for Automated Bug Fixing
- 在SWE-bench Verified基准上测试6个自动化修复系统
- 仅部分系统能解决相同漏洞，修复能力差异显著
- 适合关注AI辅助编程与缺陷修复研究者
大型语言模型（LLMs）及其基于智能体的系统已被用于自动修复软件缺陷，通过与开发环境交互、迭代验证和代码修改展现潜力。然而，对这些系统体系化分析仍有限，尤其缺乏对顶尖系统性能差异的深入探讨。本文在SWE-bench Verified基准上评估了六个修复系统，分析其整体表现，识别出所有系统均能解决或均无法解决的案例，并比较不同系统的故障定位准确率（文件级与代码符号级）及漏洞复现能力。结果表明，当前系统在模型能力与智能体流程设计方面仍需进一步优化，以提升修复有效性。
原文摘要 · Abstract (English)
Large language models (LLMs) and LLM-based Agents have been applied to fix bugs automatically, demonstrating the capability in addressing software defects by engaging in development environment interaction, iterative validation and code modification. However, systematic analysis of these agent systems remain limited, particularly regarding performance variations among top-performing ones. In this paper, we examine six repair systems on the SWE-bench Verified benchmark for automated bug fixing. We first assess each system's overall performance, noting the instances solvable by all or none of these systems, and explore the capabilities of different systems. We also compare fault localization accuracy at file and code symbol levels and evaluate bug reproduction capabilities. Through analysis, we concluded that further optimization is needed in both the LLM capability itself and the design of Agentic flow to improve the effectiveness of the Agent in bug fixing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。