46%的AI代码修复被拒绝,研究揭示其失败原因并提出改进方向。
Understanding the Rejection of Fixes Generated by Agentic Pull Requests -- Insights from the AIDev Dataset
- 分析306个未合并的AI生成补丁,归纳出14类拒因
- 近半数修复因实现错误或测试不通过被拒
- 适合关注AI编程助手落地与优化的研究者
AI编程代理在软件项目中日益用于生成修复代码的拉取请求(PR)。对AIDev数据集的初步探索发现,Copilot、Devin、Cursor和Claude等代理提出的修复中有46.41%被拒绝,造成大量人力与计算资源浪费。本文通过对306个由这些代理创建或合著的未合并PR进行定性分析,并量化其拒因。结果识别出14种拒因,归为四大类:实现错误(如不完整或方法错误)、无法通过持续集成(CI)流水线、代理无法完成实现(如无代码生成或会话丢失)、优先级过低。研究强调需在三个层面改进:指导模型选择合适修复路径、明确禁止不当方法、通过CI验证且避免引入破坏性变更。建议优化任务优先级,避免无效的人工审查与代理资源消耗。
原文摘要 · Abstract (English)
AI coding agents are increasingly used to generate pull requests (PRs) that propose code fixes in software projects. From a first exploration of the AIDev dataset, we find that 46.41\% of the fixes proposed by the agents Copilot, Devin, Cursor, and Claude are rejected. This represents a significant amount of wasted resources that require human reviews, verifications, and running tests and validations for fixes that are merely discarded. Our goal in this paper is to understand the failure modes of AI-agents, an understanding that is crucial for better integrating AI-agents as efficient teammates. In this paper, we conduct a qualitative study on a representative sample of 306 non-merged pull requests created or co-authored by the agents mentioned earlier, followed by a quantitative analysis of the reasons for rejection. Our qualitative findings identify 14 reasons divided into four high-level categories for rejecting AI-agent fixes. We observe that developers can reject fixes due to fixes whose implementation is incorrect (e.g., incomplete, wrong approach), fixes that do not pass the continuous integration (CI) pipelines and fail tests, fixes for which the agent is unable to perform the implementation (e.g., no code generated, sessions lost), and fixes whose priority is low. Our results shed light on the importance of better guiding the model at these levels: (1) proposing hints about the approach to follow for fixing an issue, (2) outlining constraints or limitations regarding the approaches that should not be taken, and (3) instructing the agent on how to validate the implementation through CI pipelines and without introducing a breaking change. Our results suggest the need for good prioritization of tasks so that generated fixes do not lead to wasted human review efforts or wasted agent resources (e.g., tokens, compute, or allowed number of requests).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。