评测大模型代码审查代理的真实效能，发现误报问题严重制约实际应用。
CR-Bench: Evaluating the Real-World Utility of AI Code Review Agents
- 构建细粒度评估框架CR-Evaluator与基准数据集CR-Bench。
- 代理在追求全检漏洞时误报率高，导致信号噪声比低。
- 揭示了修复率与误报间的权衡，适合关注工程落地的研究者。
前沿大语言模型催生了可在开放、推理密集场景中运行的代码审查代理。然而，缺乏标准化基准与细粒度评估协议，使得难以超越粗粒度成功指标来评估代理行为，尤其在误报代价高的任务中。为此，我们提出CR-Bench基准数据集和CR-Evaluator细粒度评估流程。利用该工具，我们对基于两种前沿模型的单次提示代理与基于Reflexion的代理进行了初步评估。结果表明，当设计为识别所有隐藏问题时，代码审查代理表现出低信噪比，仅以修复率衡量会掩盖真实进展与开发者生产力。分析揭示了问题修复与虚假发现之间的隐性权衡，暴露了有效代理设计的前沿瓶颈。CR-Bench与CR-Evaluator共同为研究与开发基于LLM的代码审查代理提供了及时基础，助力其从受控基准向真实软件工程工作流过渡。
原文摘要 · Abstract (English)
Recent advances in frontier large language models have enabled code review agents that operate in open-ended, reasoning-intensive settings. However, the lack of standardized benchmarks and granular evaluation protocols makes it difficult to assess behavior of code review agents beyond coarse success metrics, particularly for tasks where false positives are costly. To address this gap, we introduce CR-Bench, a benchmarking dataset, and CR-Evaluator, a fine-grained evaluation pipeline for code review agents. Using these tools, we conduct a preliminary study evaluating both a single-shot agent and a Reflexion-based agent across two frontier models. We find that code review agents can exhibit a low signal-to-noise ratio when designed to identify all hidden issues, obscuring true progress and developer productivity when measured solely by resolution rates. Our analysis identifies the hidden trade-off between issue resolution and spurious findings, revealing a frontier that constrains effective agent design. Together, CR-Bench and CR-Evaluator provide a timely foundation for studying and developing code review agents as LLM-based systems transition from controlled benchmarks to real-world software engineering workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。