首个面向多轮代码审查的动态基准,揭示大模型在真实开发中的局限。
From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench

- 构建包含2269个真实多轮审查任务的动态基准MCR-Bench
- 主流大模型在多轮交互中表现下降,缺陷检测率随轮次增加显著降低
- 发现模型存在跨轮时序错位、长程记忆不足等核心缺陷
在真实软件开发中,代码审查通常涉及开发者与评审者之间的多轮迭代互动,这一过程成本高且耗时。尽管近期研究探索了大语言模型(LLMs)在自动化代码审查中的应用,但大多数方法将审查简化为单轮静态决策任务,未能捕捉真实场景中的多轮交互特性与复杂问题解决过程。为此,我们提出MCR-Bench,首个面向缺陷状态感知的现实多轮代码审查基准。MCR-Bench覆盖五种常用编程语言,包含2,269个真实世界的多轮代码审查任务,每项任务均标注细粒度缺陷信息及跨轮状态标签。每个任务配备细粒度缺陷元数据(如描述、类型、严重性)和动态状态标注,完整记录缺陷在整个多轮过程中的演化轨迹。我们在主流大模型上对MCR-Bench进行大量实验,发现:(1) 整体能力有限:主流大模型在缺陷检测与缺陷生命周期状态追踪方面表现不佳,性能随交互轮次增加显著下降;(2) 缺陷敏感性差异:模型在不同缺陷类型和严重程度下表现差异显著,语义复杂或低显著性缺陷更易被遗漏;(3) 深层失败机制:深入误差分析揭示了误报与漏报的不同驱动因素,暴露出跨轮时序错位与长程记忆不足等关键弱点。
原文摘要 · Abstract (English)
In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming. Although recent work explores large language models (LLMs) for automated code review, most approaches oversimplify code review into a single-round, static decision task, which fails to capture the multi-round interactive nature and the complex problem-solving processes inherent in realistic review scenarios. To bridge this gap, we introduce MCR-Bench, the first defect state-aware benchmark designed for realistic multi-round code review. MCR-Bench covers five commonly-used programming languages and consists of 2,269 real-world multi-round code review tasks, each of which is annotated with fine-grained defect information and cross-round state labels. Each task in MCR-Bench is equipped with fine-grained defect metadata (e.g., description, type, severity) alongside dynamic state annotations, capturing the complete evolutionary trajectory of a defect throughout the multi-round process. We obtain several findings through extensive experiments on MCR-Bench with mainstream LLMs. (1) Limited overall capability: experiments reveal that mainstream LLMs exhibit limited overall performance in defect detection and defect lifecycle state tracking, with performance degrading significantly as the number of interaction rounds increases; (2) Defect-sensitive performance: LLMs' performance varies substantially across different defect types and severity levels, with semantically complex or low-salience defects being significantly more likely to be missed; (3) Underlying Failure Mechanisms: our in-depth error analysis dissects the distinct drivers of false positives and false negatives, revealing critical weaknesses such as cross-round temporal misalignment and inadequate long-range memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。