看似无害的提示词可暗中诱导模型生成虚假包名,带来供应链安全风险。
Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills

- 用鼓励想象等无害指令提升模型虚构包名概率
- 使虚假包名生成率和安装成功率均显著上升
- 能绕过现有各类防御机制,适合安全研究者关注
由大语言模型驱动的编码代理在软件开发中负责生成代码、选择依赖项并输出包安装命令,形成新的软件供应链风险:当代理虚构不存在的包时,攻击者可注册该名称,后续诱骗用户安装以实施攻击。现有针对包虚构的攻击与防御多聚焦于自然发生的幻觉、定向依赖引导或事后验证。本文提出一种高度隐蔽的攻击范式——中性提示攻击(Neutral Prompting Attack, NPA),即通过语义上无害的指令(如鼓励想象力、全面性)提升模型虚构包名的概率,且不包含明确恶意意图。与定向依赖引导不同,NPA不指定特定目标包,而是使模型生成更倾向推测性的包名。我们在多个面向编码的LLM及包虚构基准上评估了NPA,结果表明其显著提升了幻觉准确率(Hallucination ASR)和Pip安装准确率(Pip Install ASR),改变了虚构包名分布,并成功绕过现有的静态分析、基于LLM及代理技能类防御。这揭示出看似无害的提示词可能暗中操纵幻觉行为,引发下游软件供应链风险。
原文摘要 · Abstract (English)
LLM-powered coding agents increasingly participate in software development workflows by generating code, selecting dependencies, and producing package installation commands. This creates a new software supply chain risk: when an agent hallucinates a non-existent package, an attacker may register the hallucinated name and later compromise users who install it. Existing package hallucination attacks and defenses primarily focus on naturally occurring hallucinations, targeted dependency steering, or post-hoc package validation. In this paper, we introduce \emph{Neutral Prompting Attack} (NPA), a highly stealthy attack paradigm in which semantically benign instructions, such as encouraging imagination and exhaustiveness, increase package hallucination propensity without containing explicit malicious intent. Unlike targeted dependency steering, NPA does not specify an attacker-chosen package. Instead, it shifts the model's dependency generation behavior toward more speculative package names. We evaluate NPA across multiple coding-oriented LLMs and package hallucination benchmarks. Our results show that NPA increases both \emph{Hallucination ASR} and \emph{Pip Install ASR}, changes the distribution of hallucinated package names, and evades existing static-analysis, LLM-based, and agent-based Skill defenses. These findings reveal that harmless-looking prompts can covertly manipulate hallucination behavior and create downstream software supply chain risks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。