arXiv:2508.11257cs.SEcs.AI2025-08被引 4

研究大模型代码生成中的幻觉问题,发现其在汽车软件领域易出错。

Hallucination in LLM-Based Code Generation: An Automotive Case Study

  • 用三种提示策略测试多个大模型的代码生成能力
  • 高频率出现语法错误和接口调用错误,仅两个模型在复杂提示下正确
  • 适合关注安全关键领域代码可靠性的研究人员

大型语言模型(LLMs)在自动化代码生成方面展现出巨大潜力,但在实际应用中受限于幻觉问题——即输出看似合理却事实错误、无法验证或无意义。本文聚焦汽车领域,通过案例研究评估多个代码生成LLM在三种不同提示复杂度下的表现:从最简的一行提示,到加入Covesa车辆信号规范(VSS)上下文,再到附加代码骨架。结果表明,GPT-4.1、Codex和GPT-4o均存在高频语法违规、无效引用错误和API知识冲突。仅有GPT-4.1和GPT-4o在最丰富上下文提示下生成了正确解,而简单提示即使多次迭代优化也无法产出有效代码。这凸显了在汽车等安全关键系统中,亟需有效的幻觉缓解技术以保障代码生成的可靠性。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have shown significant potential in automating code generation tasks offering new opportunities across software engineering domains. However, their practical application remains limited due to hallucinations - outputs that appear plausible but are factually incorrect, unverifiable or nonsensical. This paper investigates hallucination phenomena in the context of code generation with a specific focus on the automotive domain. A case study is presented that evaluates multiple code LLMs for three different prompting complexities ranging from a minimal one-liner prompt to a prompt with Covesa Vehicle Signal Specifications (VSS) as additional context and finally to a prompt with an additional code skeleton. The evaluation reveals a high frequency of syntax violations, invalid reference errors and API knowledge conflicts in state-of-the-art models GPT-4.1, Codex and GPT-4o. Among the evaluated models, only GPT-4.1 and GPT-4o were able to produce a correct solution when given the most context-rich prompt. Simpler prompting strategies failed to yield a working result, even after multiple refinement iterations. These findings highlight the need for effective mitigation techniques to ensure the safe and reliable use of LLM generated code, especially in safety-critical domains such as automotive software systems.

代码生成大模型幻觉汽车软件安全关键

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。