五款前沿代码大模型仍会虚构不存在的包名，存在供应链攻击风险。
The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort
- 在19.9万条指令中测试，幻觉率压缩至4.6%-6.1%。
- 发现127个模型共有的虚构包名，53个仍可被恶意注册。
- 揭示模型间幻觉模式差异，暗示训练数据共享可能。
Spracklen等人（USENIX Security '25）发现代码生成大模型在PyPI或npm上虚构不存在包名的比例为5.2%（商业模型）至21.7%（开源模型），形成漏洞利用风险。我们复现其方法，对2025年10月至2026年3月发布的五款前沿代码模型——Claude Sonnet 4.6、Claude Haiku 4.5、GPT-5.4-mini、Gemini 2.5 Pro和DeepSeek V3.2——进行测试。在199,845组配对的Python与JavaScript提示词下，基于PyPI和npm主列表验证，整体幻觉率介于4.62%（Haiku 4.5）至6.10%（GPT-5.4-mini）之间，模型间差异显著缩小但威胁依然存在。除复现外，识别出127个被所有五款模型一致虚构的包名（PyPI 109个，npm 18个），经与PyPI Security及Socket.dev协调披露后，仍有53个（PyPI 41个，npm 12个）可被攻击者注册，构成模型无关的供应链攻击面。此外，发现反向的Python优于JavaScript幻觉倾向，颠覆了Spracklen 2024年的结论；观察到Anthropic家族中Haiku低于Sonnet的幻觉表现；并发现DeepSeek V3.2与GPT-5.4-mini之间存在Jaccard相似性峰值（J=0.343），暗示可能存在共同训练数据来源。
原文摘要 · Abstract (English)
Spracklen et al. (USENIX Security '25) showed that code-generating large language models hallucinate package names that do not exist on PyPI or npm at rates ranging from 5.2% on commercial models to 21.7% on open-source models, creating an attack surface for slopsquatting -- the registration of malicious packages under hallucinated names. We replicate their methodology on five frontier code-capable LLMs released between October 2025 and March 2026: Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4-mini, Gemini 2.5 Pro, and DeepSeek V3.2. Across 199,845 paired Python and JavaScript prompts validated against PyPI and npm master lists, we measure overall hallucination rates between 4.62% (Claude Haiku 4.5) and 6.10% (GPT-5.4-mini) -- an order-of-magnitude compression of the inter-model spread observed by Spracklen, but not a retirement of the threat. Beyond replication, we identify a set of 127 package names (109 on PyPI, 18 on npm) that all five evaluated models invent identically; following coordinated disclosure with PyPI Security and Socket.dev, 53 of these (41 on PyPI, 12 on npm) remain registrable by an attacker after each registry's existing defenses, constituting a model-agnostic supply-chain attack surface that no single-model study can reveal. We further document a Python-over-JavaScript hallucination asymmetry that inverts Spracklen's 2024 finding, identify a Haiku-below-Sonnet inversion within the Anthropic family, and observe a Jaccard-similarity peak between DeepSeek V3.2 and GPT-5.4-mini (J = 0.343) suggestive of shared training-data origins.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。