评测大模型代码审查能力，发现其纠错率超六成但仍有明显误判。
Evaluating Large Language Models for Code Review
- 用真实代码块测试GPT4o和Gemini 2.0 Flash的审查表现
- 有上下文时正确判断正确性率达68.5%，可修复代码67.8%
- 建议引入人工审核以降低错误风险，提升可靠性
代码审查对软件质量至关重要。近年来大语言模型（LLMs）已能执行代码审查与修复，已有相关工具问世，但其可靠性与准确性尚未系统评估。本研究比较了GPT4o与Gemini 2.0 Flash在检测代码正确性及提出改进建议方面的表现。实验基于492个由AI生成的、正确性不一的代码块，以及164个来自HumanEval基准的规范代码块，采用不同配置进行测试。为客观模拟代码审查任务，要求模型评估代码正确性并必要时进行优化。结果显示，在提供问题描述的情况下，GPT4o和Gemini 2.0 Flash分别在68.50%和63.89%的样本中正确判断代码正确性，并在67.83%和54.26%的样本中成功修正代码；无问题描述时性能下降。对164个规范代码块的结果显示，模型表现依赖于代码类型。结论表明，尽管大模型可辅助代码审查并提出改进，但仍存在误判风险。为此，我们提出‘人在回路的LLM代码审查’流程，兼顾知识共享与输出可靠性。
原文摘要 · Abstract (English)
Context: Code reviews are crucial for software quality. Recent AI advances have allowed large language models (LLMs) to review and fix code; now, there are tools that perform these reviews. However, their reliability and accuracy have not yet been systematically evaluated. Objective: This study compares different LLMs' performance in detecting code correctness and suggesting improvements. Method: We tested GPT4o and Gemini 2.0 Flash on 492 AI generated code blocks of varying correctness, along with 164 canonical code blocks from the HumanEval benchmark. To simulate the code review task objectively, we expected LLMs to assess code correctness and improve the code if needed. We ran experiments with different configurations and reported on the results. Results: With problem descriptions, GPT4o and Gemini 2.0 Flash correctly classified code correctness 68.50% and 63.89% of the time, respectively, and corrected the code 67.83% and 54.26% of the time for the 492 code blocks of varying correctness. Without problem descriptions, performance declined. The results for the 164 canonical code blocks differed, suggesting that performance depends on the type of code. Conclusion: LLM code reviews can help suggest improvements and assess correctness, but there is a risk of faulty outputs. We propose a process that involves humans, called the "Human in the loop LLM Code Review" to promote knowledge sharing while mitigating the risk of faulty outputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。