分析AI生成修复代码的合并情况，发现测试失败和重复修复是主因。
Why Are AI Agent Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study
- 基于8106个AI生成的修复PR，统计合并与未合并比例。
- 326个未合并PR中，测试失败占多数，重复修复次之。
- 揭示当前AI编程工具在真实开发中的关键短板。
自主编程代理（如OpenAI Codex、Devin、GitHub Copilot）在真实软件仓库中越来越多地生成修复类拉取请求（PR）。但其实际有效性取决于项目维护者是否接受并合并这些贡献。本文对由五个广泛使用的AI编程代理生成的8,106个修复类PR进行实证研究，量化了被合并、关闭未合并或仍开放的占比。随后，对326个关闭但未合并的PR进行人工定性分析，耗时约100人小时，构建出包含12种失败原因的结构化目录。结果表明，测试用例失败和同一问题已被其他PR解决是导致未集成的最常见原因，而构建或部署失败相对较少。总体而言，研究揭示了当前AI编程代理在真实场景下的关键局限，并指明了改进方向及更有效的人机协作路径。
原文摘要 · Abstract (English)
Autonomous coding agents (e.g., OpenAI Codex, Devin, GitHub Copilot) are increasingly used to generate fix-related pull requests (PRs) in real world software repositories. However, their practical effectiveness depends on whether these contributions are accepted and merged by project maintainers. In this paper, we present an empirical study of AI agent involved fix related PRs, examining both their integration outcomes, latency, and the factors that hinder successful merging. We first analyze 8,106 fix related PRs authored by five widely used AI coding agents from the AIDEV POP dataset to quantify the proportions of PRs that are merged, closed without merging, or remain open. We then conduct a manual qualitative analysis of a statistically significant sample of 326 closed but unmerged PRs, spending approximately 100 person hours to construct a structured catalog of 12 failure reasons. Our results indicate that test case failures and prior resolution of the same issues by other PRs are the most common causes of non integration, whereas build or deployment failures are comparatively rare. Overall, our findings expose key limitations of current AI coding agents in real world settings and highlight directions for their further improvement and for more effective human AI collaboration in software maintenance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。