arXiv:2511.07017cs.SEcs.AI2025-11综述被引 7

构建细粒度代码审查基准,融合上下文信息提升LLM评估可靠性

Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice

  • 引入文本与代码双重上下文,增强对开发意图的理解
  • 经多阶段过滤后保留67,910个高质量样本,支持行级审查评估
  • 实测表明文本上下文比代码上下文更重要,且仍远低于人工水平

代码审查是保障软件质量的核心环节,近年来大语言模型(LLMs)在自动化代码审查方面展现出潜力。然而,现有基于LLM的代码审查基准存在三大局限:缺乏语义上下文,多数仅提供代码差异而无问题描述等文本信息;数据质量不佳,未经过严格验证的样本常含过时或无关代码,降低评估可信度;粒度粗略,普遍以文件或提交为单位,忽略关键的行级推理。为此,本文提出ContextCRBench,一个高质量、上下文丰富的细粒度代码审查评估基准。其构建流程包括:原始数据爬取,从顶级仓库收集153.7万条问题与拉取请求;全面上下文提取,关联问题-拉取请求对获取文本上下文,并提取完整函数或类作为代码上下文;多阶段数据过滤,结合规则与LLM验证剔除过时、错误或低价值样本,最终获得67,910个上下文丰富样本。ContextCRBench支持三种符合实际审查流程的评估场景:块级质量评估、行级缺陷定位与行级评论生成。对八款主流模型(四款闭源、四款开源)的评测显示,文本上下文带来的性能提升大于仅依赖代码上下文,但当前模型仍显著落后于人类水平。该基准已在字节跳动部署,驱动自进化代码审查系统,性能提升61.98%,验证了其鲁棒性与工业实用性。

原文摘要 · Abstract (English)

Code review is a cornerstone of software quality assurance, and recent advances in Large Language Models (LLMs) have shown promise in its automation. However, existing benchmarks for LLM-based code review face three major limitations. Lack of semantic context: most benchmarks provide only code diffs without textual information such as issue descriptions, which are crucial for understanding developer intent. Data quality issues: without rigorous validation, many samples are noisy-e.g., reviews on outdated or irrelevant code-reducing evaluation reliability. Coarse granularity: most benchmarks operate at the file or commit level, overlooking the fine-grained, line-level reasoning essential for precise review. We introduce ContextCRBench, a high-quality, context-rich benchmark for fine-grained LLM evaluation in code review. Our construction pipeline comprises: Raw Data Crawling, collecting 153.7K issues and pull requests from top-tier repositories; Comprehensive Context Extraction, linking issue-PR pairs for textual context and extracting the full surrounding function or class for code context; and Multi-stage Data Filtering, combining rule-based and LLM-based validation to remove outdated, malformed, or low-value samples, resulting in 67,910 context-enriched entries. ContextCRBench supports three evaluation scenarios aligned with the review workflow: hunk-level quality assessment, line-level defect localization, and line-level comment generation. Evaluating eight leading LLMs (four closed-source and four open-source) reveals that textual context yields greater performance gains than code context alone, while current LLMs remain far from human-level review ability. Deployed at ByteDance, ContextCRBench drives a self-evolving code review system, improving performance by 61.98% and demonstrating its robustness and industrial utility. https://github.com/kinesiatricssxilm14/ContextCRBench.

代码审查大模型评估上下文增强工业应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。