分析2.4万份AI生成代码请求，发现其修改方式与人类显著不同。
How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests
- 对比2.4万份AI与5千份人类提交的代码变更模式
- AI提交的改动条数更多，文件和删除行变化中等差异
- AI的修改描述与代码变动更匹配，适合评估其可靠性
AI编程代理正越来越多地以自主贡献者身份生成并提交拉取请求（PR）。然而，我们缺乏关于这些代理生成的PR与人类贡献在代码修改方式及变更描述上差异的实证证据。理解这些差异对评估其可靠性和对开发流程的影响至关重要。本研究基于MSR 2026 Mining Challenge版本的AIDev数据集，分析了24,014个已合并的代理类PR（共440,295次提交）和5,081个已合并的人类PR（共23,242次提交）。研究考察了代码增删、提交次数和触及文件数，并通过词法与语义相似性评估了PR描述与代码差异（diff）的一致性。结果显示，代理类PR在提交次数上与人类有显著差异（Cliff's $δ= 0.5429$），在触及文件数和删除行数上呈现中等差异；同时，其描述与代码变动在各项指标上的相似性略高。该研究提供了对AI编程代理在开源开发中贡献行为的大规模实证刻画。
原文摘要 · Abstract (English)
AI coding agents are increasingly acting as autonomous contributors by generating and submitting pull requests (PRs). However, we lack empirical evidence on how these agent-generated PRs differ from human contributions, particularly in how they modify code and describe their changes. Understanding these differences is essential for assessing their reliability and impact on development workflows. Using the MSR 2026 Mining Challenge version of the AIDev dataset, we analyze 24,014 merged Agentic PRs (440,295 commits) and 5,081 merged Human PRs (23,242 commits). We examine additions, deletions, commits, and files touched, and evaluate the consistency between PR descriptions and their diffs using lexical and semantic similarity. Agentic PRs differ substantially from Human PRs in commit count (Cliff's $δ= 0.5429$) and show moderate differences in files touched and deleted lines. They also exhibit slightly higher description-to-diff similarity across all measures. These findings provide a large-scale empirical characterization of how AI coding agents contribute to open source development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。