arXiv:2603.26130cs.SEcs.AI2026-03综述被引 2

8个前沿模型在代码审查中仅能发现15%-31%的人工标记问题,远低于人类水平。

SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback

  • 基于350个真实PR构建基准,用人类标注作为黄金标准评估AI代码审查效果。
  • 在仅提供diff的情况下,所有模型检测率仅为15%-31%,且上下文越长表现越差。
  • 即使加入结构化语义信息,模型仍难以识别上下文相关缺陷,适合关注代码审查局限性的研究者。

我们提出SWE-PRBench,一个包含350个拉取请求的基准数据集,配有经人工标注的真实标签,用于评估AI代码审查质量。在经过验证的LLM-as-judge框架(kappa=0.75)下评估,8个前沿模型在仅提供diff的配置下,仅能检测到15%-31%的人工标记问题,表明尽管在代码生成任务中表现优异,但当前AI代码审查仍远落后于人类专家。数据源自活跃开源仓库,从700个候选中通过仓库质量评分筛选,并在三种冻结上下文配置下评估:仅diff(config_A)、diff加文件内容(config_B)、完整上下文(config_C),支持系统性消融分析。所有模型从config_A到config_C均呈单调下降趋势,即使引入AST提取函数上下文和导入图解析等结构化语义层亦无改善。主要问题是Type2_Contextual问题在config_B中检测能力崩溃,与长上下文注意力稀释一致:一个2,000词的diff+摘要提示优于包含执行上下文、行为映射和测试签名的2,500词全上下文提示。前四名模型得分相近(平均0.147-0.153),后四名明显更低(平均≤0.113)。数据集、上下文、标注及评估工具已公开发布。

原文摘要 · Abstract (English)

We introduce SWE-PRBench, a benchmark of 350 pull requests with human-annotated ground truth for evaluating AI code review quality. Evaluated against an LLM-as-judge framework validated at kappa=0.75, 8 frontier models detect only 15-31% of human-flagged issues on the diff-only configuration, demonstrating that AI code review remains far below human expert performance despite strong results on code generation benchmarks. Pull requests are drawn from active open-source repositories, filtered from 700 candidates using a Repository Quality Score, and evaluated under three frozen context configurations: diff only (config_A), diff with file content (config_B), and full context (config_C), enabling systematic ablation of context provision strategies. All 8 models degrade monotonically from config_A to config_C, even when context is provided via structured semantic layers including AST-extracted function context and import graph resolution. The dominant mechanism is a collapse of Type2_Contextual issue detection at config_B, consistent with attention dilution in long contexts: a structured 2,000-token diff-with-summary prompt outperforms a 2,500-token full-context prompt enriched with execution context, behaviour mapping, and test signatures across all 8 models. The top four models are statistically indistinguishable (mean score 0.147-0.153) while a clear tier gap separates them from the remaining four (mean score <= 0.113). Dataset, contexts, annotations, and evaluation harness are released publicly.

代码审查基准测试LLM评估AI质量

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。