AI生成的代码提交说明与实际代码不符，影响人类评审信任度。
Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requests
- 通过分析2.3万条AI提交，发现1.7%存在描述与代码不一致
- 高不一致提交接受率仅28.3%，合并时间长达55.8小时
- 最常见问题是描述声称实现未完成的改动，占比45.4%
AI编码代理生成的拉取请求（PR）描述是向人类评审者传达代码变更的主要渠道。然而，这些描述与实际代码变更之间的对齐关系尚未被充分研究，引发了对AI代理可信度的担忧。为填补这一空白，我们分析了五个代理在23,247个代理型PR中产生的PR消息-代码不一致（PR-MCI）。我们贡献了974个手动标注的PR，发现406个PR（1.7%）表现出高PR-MCI，并识别出八种类型，其中“描述声称未实现的变更”最为普遍（45.4%）。统计检验表明，高MCI PR的接受率比低MCI PR低51.7%（28.3% vs. 80.0%），合并时间延长3.5倍（55.8小时 vs. 16.0小时）。研究结果表明，不可靠的PR描述削弱了对AI代理的信任，凸显了引入PR-MCI验证机制和改进PR生成的必要性，以实现可信的人机协作。
原文摘要 · Abstract (English)
Pull request (PR) descriptions generated by AI coding agents are the primary channel for communicating code changes to human reviewers. However, the alignment between these messages and the actual changes remains unexplored, raising concerns about the trustworthiness of AI agents. To fill this gap, we analyzed 23,247 agentic PRs across five agents using PR message-code inconsistency (PR-MCI). We contributed 974 manually annotated PRs, found 406 PRs (1.7%) exhibited high PR-MCI, and identified eight PR-MCI types, revealing that "descriptions claim unimplemented changes" was the most common issue (45.4%). Statistical tests confirmed that high-MCI PRs had 51.7% lower acceptance rates (28.3% vs. 80.0%) and took 3.5 times longer to merge (55.8 vs. 16.0 hours). Our findings suggest that unreliable PR descriptions undermine trust in AI agents, highlighting the need for PR-MCI verification mechanisms and improved PR generation to enable trustworthy human-AI collaboration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。