分析3.3万条AI提交的代码请求,发现文档类任务最易通过,性能优化类最易失败。
Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub
- 对比3.3万条AI生成的代码请求,按任务类型、代码量、测试结果等维度量化分析
- 文档和构建更新类任务合并率最高,性能修复类任务失败率超70%且常不通过测试
- 发现AI误判需求、重复提交、评审无反馈等关键失败模式,适合研究人机协作的开发者
AI编码代理正越来越多地向开源项目提交代码请求(PR),作为自主贡献者而非仅辅助角色。随着这类代理贡献在真实仓库中迅速增长,我们对其实际表现及失败原因知之甚少。本文对五个编码代理在GitHub上提交的3.3万条代理生成的PR进行了大规模研究。研究首先从任务类型、代码变更、CI构建结果和评审动态四个维度定量分析已合并与未合并的PR。结果显示,文档、CI和构建更新类任务合并成功率最高,而性能优化与缺陷修复类任务表现最差;未合并的PR通常涉及更大的代码变更、触及更多文件,且多数无法通过项目的CI/CD验证。为进一步探究失败原因,我们定性分析了600个PR,提炼出一套分层的拒绝模式分类体系。该分析揭示了定量指标未能涵盖的拒因,包括缺乏有意义的评审互动、重复提交、非预期功能实现以及代理目标错位。综合发现表明,社会-技术因素和人-智能协作机制是提升未来代理工作流成功的关键。
原文摘要 · Abstract (English)
AI coding agents are now submitting pull requests (PRs) to software projects, acting not just as assistants but as autonomous contributors. As these agentic contributions are rapidly increasing across real repositories, little is known about how they behave in practice and why many of them fail to be merged. In this paper, we conduct a large-scale study of 33k agent-authored PRs made by five coding agents across GitHub. (RQ1) We first quantitatively characterize merged and not-merged PRs along four broad dimensions: 1) merge outcomes across task types, 2) code changes, 3) CI build results, and 4) review dynamics. We observe that tasks related to documentation, CI, and build update achieve the highest merge success, whereas performance and bug-fix tasks perform the worst. Not-merged PRs tend to involve larger code changes, touch more files, and often do not pass the project's CI/CD pipeline validation. (RQ2) To further investigate why some agentic PRs are not merged, we qualitatively analyze 600 PRs to derive a hierarchical taxonomy of rejection patterns. This analysis complements the quantitative findings in RQ1 by uncovering rejection reasons not captured by quantitative metrics, including lack of meaningful reviewer engagement, duplicate PRs, unwanted feature implementations, and agent misalignment. Together, our findings highlight key socio-technical and human-AI collaboration factors that are critical to improving the success of future agentic workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。