首个专为AI代码审查代理设计的基准测试数据集，评估其审查能力。
Code Review Agent Benchmark
- 构建c-CRAB数据集，基于人工评审生成可验证的代码审查任务。
- 当前主流代理仅能解决约40%的任务，存在显著提升空间。
- 适合研究代码生成与审查协同、人机协作的AI研发团队。
随着AI代理在代码生成中展现巨大潜力，自动生成代码的规模迅速增长，代码质量问题日益突出。当这些代码被集成到大型代码库时，代码审查与质量保障变得至关重要。本文提出一个全新的代码审查基准数据集c-CRAB（发音如see-crab），用于评估代码审查代理的能力。给定一个来自代码生成代理或人类的拉取请求（Pull Request），该框架可评估代码审查代理生成的评论质量。我们使用c-CRAB对当前主流的开源代理PR-agent及商业工具Devin、Claude Code、Codex进行了评测。数据集通过人工评审实例生成对应的测试用例，实现对代理评审结果的系统性评估。结果显示，现有代理整体仅能完成约40%的任务，表明仍有巨大改进空间；且代理的评审视角常与人类不同，提示未来可开展人机协同审查。此外，代理生成的测试用例可作为独立测试集，充当审查质量的“质量门控”，为代码生成、测试生成与审查代理的协同研究提供基础。
原文摘要 · Abstract (English)
Software engineering agents have shown significant promise in writing code. As AI agents permeate code writing, and generate huge volumes of code automatically -- the matter of code quality comes front and centre. As the automatically generated code gets integrated into huge code-bases -- the issue of code review and broadly quality assurance becomes important. In this paper, we take a fresh look at the problem and curate a code review dataset for AI agents to work with. Our dataset called c-CRAB (pronounced see-crab) can evaluate agents for code review tasks. Specifically given a pull-request (which could be coming from code generation agents or humans), if a code review agent produces a review, our evaluation framework can asses the reviewing capability of the code review agents. Our evaluation framework is used to evaluate the state of the art today -- the open-source PR-agent, as well as commercial code review agents from Devin, Claude Code, and Codex. Our c-CRAB dataset is systematically constructed from human reviews -- given a human review of a pull request instance we generate corresponding tests to evaluate the code review agent generated reviews. Such a benchmark construction gives us several insights. Firstly, the existing review agents taken together can solve only around 40% of the c-CRAB tasks, indicating the potential to close this gap by future research. Secondly, we observe that the agent reviews often consider different aspects from the human reviews -- indicating the potential for human-agent collaboration for code review that could be deployed in future software teams. Last but not the least, the agent generated tests from our data-set act as a held out test-suite and hence quality gate for agent generated reviews. What this will mean for future collaboration of code generation agents, test generation agents and code review agents -- remains to be investigated.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。