arXiv:2508.03622cs.AI2025-08

针对大模型代码生成中错误前提导致的幻觉问题,提出首个专项评估框架。

Refining Critical Thinking in LLM Code Generation: A Faulty Premise-based Evaluation Framework

  • 构建三类错误前提,系统测试大模型在异常输入下的表现
  • 发现多数模型依赖外部提示才能纠错,自我审查能力弱
  • 揭示不同错误类型触发不同缺陷模式,为改进提供方向

随着大语言模型(LLMs)代码生成能力的提升,其对输入前提的依赖性日益增强。当用户输入包含错误前提时,代码生成幻觉概率显著上升,暴露出模型自我审查能力的不足。本文提出首个针对错误前提的代码生成评估框架——故障前提基准(FPBench)。通过系统构建三类错误前提,并集成多维评估指标,对15个代表性大模型进行了深入评估。主要发现:(1) 多数模型在错误前提下推理能力差,代码生成性能不佳,严重依赖显式提示进行错误检测,自我审查能力有限;(2) 错误前提引发资源投入的收益递减点,盲目增加长度无法提升质量;(3) 三类错误前提分别激活模型中不同的缺陷模式,揭示了代码生成模型认知机制的三重分离性。本研究不仅强调大模型在代码生成中主动验证前提的紧迫性,也通过提出的FPBench框架与多维评估体系,为发展可靠、以人为本的代码生成模型提供了理论基础与实践路径。

原文摘要 · Abstract (English)

With the advancement of code generation capabilities in large language models (LLMs), their reliance on input premises has intensified. When users provide inputs containing faulty premises, the probability of code generation hallucinations rises significantly, exposing deficiencies in their self-scrutiny capabilities. This paper proposes Faulty Premises Bench (FPBench), the first code generation evaluation framework targeting faulty premises. By systematically constructing three categories of faulty premises and integrating multi-dimensional evaluation metrics, it conducts in-depth assessments of 15 representative LLMs. The key findings are as follows: (1) Most models exhibit poor reasoning abilities and suboptimal code generation performance under faulty premises, heavily relying on explicit prompts for error detection, with limited self-scrutiny capabilities; (2) Faulty premises trigger a point of diminishing returns in resource investment, leading to blindly increasing length fails to enhance quality; (3) The three types of faulty premises respectively activate distinct defect patterns in models, revealing a triple dissociation in the cognitive mechanisms of code generation models. This study not only highlights the urgent need for LLMs to proactively verify premises in code generation but also, through the proposed FPBench framework and multi-dimensional evaluation system, provides a theoretical foundation and practical pathway for developing reliable, human-centric code generation models.

代码生成大模型评估错误检测认知机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。