分析开发智能体在真实GitHub问题中的纠错过程,揭示其失败原因与调试难点。
Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios
- 通过追踪3977个求解轨迹,研究智能体解决实际代码问题的动态过程。
- 发现执行错误如ModuleNotFoundError会降低修复率并增加推理负担。
- 揭示平台缺陷并公开数据,助力公平评估与可复现研究。
以大语言模型为基础的软件开发智能体已能完成复杂的仓库级任务,不仅生成最终代码,还进行多步推理、调用工具修改代码、调试并交互式诊断问题。然而,现有评估大多聚焦于最终代码的静态分析,难以反映其动态求解过程。为此,我们对500个GitHub问题上8个顶级智能体的3,977个求解阶段轨迹和3,931个测试阶段日志进行了深入实证研究。结果表明,求解阶段的Python执行错误(如ModuleNotFoundError、TypeError)与较低的修复率及更高的推理开销显著相关。尤其严重的错误包括OSError和数据库相关错误(如IntegrityError),需更多调试努力。此外,我们发现了SWE-Bench平台中的3个影响评估公平性与准确性的缺陷,并已报告且获得维护者确认。为推动透明化研究,我们公开了全部数据集与分析脚本。
原文摘要 · Abstract (English)
AI-driven software development has rapidly advanced with the emergence of software development agents that leverage large language models (LLMs) to tackle complex, repository-level software engineering tasks. These agents go beyond just generation of final code; they engage in multi-step reasoning, utilize various tools for code modification and debugging, and interact with execution environments to diagnose and iteratively resolve issues. However, most existing evaluations focus primarily on static analyses of final code outputs, yielding limited insights into the agents' dynamic problem-solving processes. To fill this gap, we conduct an in-depth empirical study on 3,977 solving-phase trajectories and 3,931 testing-phase logs from 8 top-ranked agents evaluated on 500 GitHub issues in the SWE-Bench benchmark. Our exploratory analysis shows that Python execution errors during the issue resolution phase correlate with lower resolution rates and increased reasoning overheads. We have identified the most prevalent errors -- such as ModuleNotFoundError and TypeError -- and highlighted particularly challenging errors like OSError and database-related issues (e.g., IntegrityError) that demand significantly more debugging effort. Furthermore, we have discovered 3 bugs in the SWE-Bench platform that affect benchmark fairness and accuracy; these issues have been reported to and confirmed by the maintainers. To promote transparency and foster future research, we publicly share our datasets and analysis scripts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。