LLM代码审查常误判正确代码,越详细的提示反而错误越多。
Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgement
- 用自然语言需求测试代码,发现模型频繁把正确代码判为错误。
- 要求解释和修复建议的提示使误判率上升,暴露可靠性缺陷。
- 提出基于修复方案的验证过滤器,提升审查可信度,适合开发者参考。
大型语言模型(LLMs)已成为软件开发中需求工程、代码生成与审查的重要工具。工程师常依赖其判断代码是否满足任务描述,以确保代码稳健性与准确性。然而,现有研究尚不明确LLMs能否可靠地将代码与自然语言需求匹配。本文通过广泛采用的基准和统一提示设计,揭示了LLMs在匹配代码与自然语言需求时存在系统性失败:频繁将正确实现误判为不合规或有缺陷。令人意外的是,更详细的提示(尤其是要求提供解释和修复建议)反而导致更高的误判率,凸显基于LLM的代码助手的重大可靠性问题。我们进一步分析了故障机制,并评估了需理由判断的可靠性。基于此,提出一种修复引导的验证过滤器(Fix-guided Verification Filter),将模型提出的修复视为可执行的反事实证据,利用基准测试与规范约束增强测试对原版和修订版代码进行验证。结果揭示了此前未被充分关注的基于LLM的代码审查局限,为在自动化审查与开发流水线中集成带防护机制的LLM评审提供了实践指导。
原文摘要 · Abstract (English)
Large language models (LLMs) have become essential tools in software development, widely used for requirements engineering, code generation and review tasks. Software engineers often rely on LLMs to verify if code implementation satisfy task requirements, thereby ensuring code robustness and accuracy. However, it remains unclear whether LLMs can reliably determine code against the given task descriptions, which is usually in a form of natural language specifications. In this paper, we uncover a systematic failure of LLMs in matching code to natural language requirements. Specifically, with widely adopted benchmarks and unified prompts design, we demonstrate that LLMs frequently misclassify correct code implementation as non-compliant or defective. Surprisingly, we find that more detailed prompt design, particularly with those requiring explanations and proposed corrections, leads to higher misjudgment rates, highlighting critical reliability issues for LLM-based code assistants. We further analyze the mechanisms driving these failures and evaluate the reliability of rationale-required judgments. Building on these findings, we propose a Fix-guided Verification Filter that treats the model proposed fix as executable counterfactual evidence, and validates the original and revised implementations using benchmark tests and spec-constrained augmented tests. Our results expose previously under-explored limitations in LLM-based code review capabilities, and provide practical guidance for integrating LLM-based reviewers with safeguards in automated review and development pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。