arXiv:2505.20749cs.AIcs.SE2025-05中稿 · NeurIPS被引 7

测试发现现有编程代理解决智能体问题能力极弱,仅0.67%~4.67%成功率。

Can Agents Fix Agent Issues?

  • 构建50个可复现的智能体问题任务,含执行环境与故障测试用例
  • 实测主流软件工程代理在真实智能体问题上解决率不足5%
  • 揭示智能体维护比传统软件更难,需专门研究新方法

基于大语言模型的智能体系统正成为新兴软件范式,广泛应用于医疗、机器人和编程等领域。然而,这些系统因易出错且持续演化,维护成本极高。自动修复智能体问题(如缺陷报告或功能请求)是关键挑战。尽管近期软件工程代理(如SWE-agent)在传统软件中表现良好,但其对智能体系统的适用性仍不明确。为此,我们人工分析了201个真实智能体问题,识别出常见问题类别。随后投入500人时构建AgentIssue-Bench基准,包含50个可执行环境和故障测试的任务。评估结果显示,顶尖软件工程代理在该基准上的问题解决率仅为0.67%至4.67%。这一结果凸显了智能体系统维护的独特挑战,亟需发展专用的先进软件工程代理。数据与代码已开源。

原文摘要 · Abstract (English)

LLM-based agent systems are emerging as a new software paradigm and have been widely adopted across diverse domains such as medicine, robotics, and programming. However, maintaining these systems requires substantial effort, as they are inevitably prone to bugs and continually evolve to meet changing external requirements. Therefore, automatically resolving agent issues (i.e., bug reports or feature requests) is a crucial and challenging task. While recent software engineering (SE) agents (e.g., SWE-agent) have shown promise in addressing issues in traditional software systems, it remains unclear how effectively they can resolve real-world issues in agent systems, which differ significantly from traditional software. To fill this gap, we first manually analyze 201 real-world agent issues and identify common categories of agent issues. We then spend 500 person-hours constructing AgentIssue-Bench, a reproducible benchmark comprising 50 agent issue resolution tasks (each with an executable environment and failure-triggering tests). We further evaluate state-of-the-art SE agents on AgentIssue-Bench and reveal their limited effectiveness (i.e., with only 0.67% - 4.67% resolution rates). These results underscore the unique challenges of maintaining agent systems compared to traditional software, highlighting the need for further research to develop advanced SE agents for resolving agent issues. Data and code are available at https://github.com/alfin06/AgentIssue-Bench.

智能体软件工程自动化修复

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。