arXiv:2509.22202cs.SEcs.CL2025-09中稿 · EMNLP被引 3

研究大模型生成代码时虚构库的风险,发现微小输入变化就易引发严重错误。

Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queries

  • 分析真实开发者提问变体对大模型造库行为的影响。
  • 一字符拼写错误导致26%任务出现虚构库,假库名被接受率达99%。
  • 适合关注AI生成代码安全性的开发者与工具设计者阅读。

大型语言模型在代码生成中扮演核心角色,但仍频繁虚构不存在的库。这类库幻觉不仅是无害错误,更可能误导开发者、破坏构建流程,并引发供应链攻击如垃圾包劫持。尽管风险日益受关注,但对其在真实使用场景下的表现仍缺乏系统理解。为此,我们首次系统研究用户提示变化如何影响大模型生成代码中的库幻觉。在七种不同大模型上,分析无效导入(库名幻觉)和有效库中的无效调用(成员幻觉),考察真实开发者语言及受控错误(如拼写错误、虚构库或成员)。结果揭示系统性漏洞:单字符拼写错误可引发26%任务的幻觉;虚构库名被接受比例高达99%;时间相关提示诱发85%的幻觉。基于高风险提示,我们提出LibHalluBench基准,支持对库幻觉的系统化、可复现评估。研究凸显大模型对自然提示变化的脆弱性,强调亟需防范库相关幻觉及其下游风险。

原文摘要 · Abstract (English)

Large language models (LLMs) now play a central role in code generation, yet they continue to hallucinate, frequently inventing non-existent libraries. Such library hallucinations are not just benign errors: they can mislead developers, break builds, and expose systems to supply chain threats such as slopsquatting. Despite growing awareness of these risks, there is limited understanding of how library hallucinations manifest under realistic usage conditions. To fill this gap, we present the first systematic study of how user-level prompt variations influence library hallucinations in LLM-generated code. Across seven diverse LLMs, we analyse library name hallucinations (invalid imports) and library member hallucinations (invalid calls from valid libraries), examining the effects of realistic developer language and controlled user mistakes, including misspellings and fabricated libraries or members. Our findings expose systemic vulnerabilities: one-character misspellings trigger hallucinations in up to 26% of tasks; fabricated library names are accepted in up to 99%; and time-based prompts induce hallucinations in up to 85%. Grounded in the highest-risk prompts identified in our study, we introduce LibHalluBench, a benchmark that enables a systematic and reproducible evaluation of these library hallucinations. Our findings underscore the fragility of LLMs to natural prompt variation and highlight the urgent need for safeguards against library-related hallucinations and their downstream risks.

代码生成幻觉检测大模型安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。