arXiv:2603.27524cs.SEcs.AI2026-03中稿 · 23rd International…

AI写代码比人少引版本冲突,但改代码时更易出错。

Safer Builders, Risky Maintainers: A Comparative Study of Breaking Changes in Human vs Agentic PRs

  • 用抽象语法树分析代码变更,识别潜在破坏性修改
  • 生成任务中AI引入破坏性变更率3.45%,低于人类的7.40%
  • 维护类修改中AI风险更高,信心高也不代表安全

AI编程代理正日益融入现代软件工程流程,与人类开发者协作在开源仓库中提交代码合并请求(PR)。尽管代理提升了开发效率,其生成的代码常比人工代码包含更多缺陷和安全问题。虽然人工PR常导致向后兼容性破坏,但代理引发此类问题的潜力尚未被充分研究。本文通过对比分析来自AIDev数据集的7,191个代理生成与1,402个人工编写Python项目中的PR,评估AI PR的可靠性。我们开发了一款工具,基于抽象语法树(AST)分析对应提交的代码变化,检测潜在的破坏性变更。结果表明,在代码生成任务中,代理引入破坏性变更的比例(3.45%)显著低于人类(7.40%)。然而,在维护任务中,重构和琐碎修改分别引入破坏性变更6.72%和9.35%,风险远高于生成任务。我们还发现‘信心陷阱’现象:高置信度的代理PR仍频繁引入破坏性变更,提示在维护类变更中无论置信度高低都需严格审查。

原文摘要 · Abstract (English)

AI coding agents are increasingly integrated into modern software engineering workflows, actively collaborating with human developers to create pull requests (PRs) in open-source repositories. Although coding agents improve developer productivity, they often generate code with more bugs and security issues than human-authored code. While human-authored PRs often break backward compatibility, leading to breaking changes, the potential for agentic PRs to introduce breaking changes remains underexplored. The goal of this paper is to help developers and researchers evaluate the reliability of AI-generated PRs by examining the frequency and task contexts in which AI agents introduce breaking changes. We conduct a comparative analysis of 7,191 agent-generated PRs with 1402 human-authored PRs from Python repositories in the AIDev dataset. We develop a tool that analyzes code changes in commits corresponding to the agentic PRs and leverages an abstract syntax tree (AST) based analysis to detect potential breaking changes. Our findings show that AI agents introduce fewer breaking changes overall than humans (3.45% vs. 7.40%) in code generation tasks. However, agents exhibit substantially higher risk during maintenance tasks, with refactoring and chore changes introducing breaking changes at rates of 6.72% and 9.35%, respectively. We also identify a "Confidence Trap" where highly confident agentic PRs still introduce breaking changes, indicating the need for stricter review during maintenance oriented changes regardless of reported confidence score.

AI编程代码安全维护风险

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。