实证分析代码审查代理对开发者反馈的影响,发现多数建议被拒。
Is Agentic Code Review Helpful? Mining Developers' Feedback to CodeRabbit Reviews in the Wild

- 通过分析3.1万条代码评审与反馈,研究代理评审效果
- 36.4%建议被接受,56.3%被拒绝,主因是误报或不符开发意图
- 轻量级模型可预测拒绝率,达76%准确率,适合优化评审系统
自主代码审查代理(如CodeRabbit)正被集成到开发流程中,但其在真实场景中的实际效果尚缺乏实证。本文以CodeRabbit为例,基于239个GitHub仓库的10,191个PR,分析了31,073对代码评审与开发者反馈。结果表明:36.4%的代理评审被接受,7.3%引发讨论,56.3%被拒绝。拒绝主因包括误报、冗余、超范围建议,以及与开发者意图和编码习惯不一致。此外,代理更关注功能问题,但此类建议更容易无效。我们探索了多种基于LLM的拒绝预测方法,发现轻量级学习模型可达到76% F1分数,表明存在可学习的评审反馈模式。研究揭示了当前代码审查代理的改进空间与实践障碍。
原文摘要 · Abstract (English)
Agentic code review, where autonomous agents provide code review comments on pull requests, is increasingly integrated into development workflows, yet there is limited empirical evidence on how developers respond to such comments in practice. In this paper, we present an empirical study of agentic code reviews using CodeRabbit as a case study. Through an empirical study of 31,073 pairs of code reviews and developer feedback from 10,191 pull requests across 239 GitHub repositories, our results show that agentic reviews receive mixed reception: 36.4% were accepted and 7.3% triggered discussion, while 56.3% were rejected. Rejections were primarily associated with invalid suggestions that were false positives, redundant, or out of scope, as well as misalignment with developer intent and coding practices. We further found that agentic reviews tend to focus more on functional concerns than evolvability-related comments, yet they were more likely to be invalid. To improve effectiveness in review practices, we explored various LLM-based approaches for predicting review rejection. We found that lightweight learning-based methods achieve up to 76% F1 score, suggesting learnable patterns exist between code reviews and their corresponding feedback. Our results highlight the current state of CodeRabbit's agentic code reviews, showing opportunity gaps for improvement, as well as shortcomings hindering its effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。