分析大模型代码生成错误类型及原因，提升代码质量
A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why?
- 通过人工分析识别出7类非语法错误，4类被以往研究忽略
- 发现错误主因包括误导性函数签名等6种情况，其中误读接口最常见
- 验证GPT-4结合ReAct提示可识别65%的错误原因，适合开发者调试
大型语言模型（LLMs）在自动化代码生成中广泛应用，但仍可能生成不符合规范的缺陷代码。现有研究多关注独立函数生成错误，忽视了真实开发中依赖外部环境等上下文的影响。本文同时考察了孤立函数与依赖上下文两种生成场景，通过大量人工分析识别出七类非语法错误，其中四类为先前工作所遗漏。我们从多个角度提出六类导致错误的原因。进一步评估发现，采用ReAct提示技术的GPT-4在识别错误原因时可达到最高0.65的F1分数，如对误导性函数签名的理解偏差。这些发现有助于提升大模型生成代码的质量。
原文摘要 · Abstract (English)
Recent advancements in Large Language Models (LLMs) have led to their widespread application in automated code generation. However, these models can still generate defective code that deviates from the specification. Previous research has mainly focused on the mistakes in LLM-generated standalone functions, overlooking real-world software development situations where the successful generation of the code requires software contexts such as external dependencies. In this paper, we considered both of these code generation situations and identified a range of \textit{non-syntactic mistakes} arising from LLMs' misunderstandings of coding question specifications. Seven categories of non-syntactic mistakes were identified through extensive manual analyses, four of which were missed by previous works. To better understand these mistakes, we proposed six reasons behind these mistakes from various perspectives. Moreover, we explored the effectiveness of LLMs in detecting mistakes and their reasons. Our evaluation demonstrated that GPT-4 with the ReAct prompting technique can achieve an F1 score of up to 0.65 when identifying reasons for LLM's mistakes, such as misleading function signatures. We believe that these findings offer valuable insights into enhancing the quality of LLM-generated code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。