让大模型误判指令来源，攻击者只需模仿用户语气即可成功
Prompt Injection as Role Confusion
- 通过角色感知探测发现模型凭语调判断发言者身份
- 零样本攻击使前沿模型中60%的推理被伪造，基线几乎为零
- 攻击成功率可提前预测，适合研究安全与防御的学者
大语言模型将世界视为单一文本流，按<user>或<tool>等角色划分。我们发现提示注入源于角色混淆：模型根据语调而非标签判断文本来源。隐藏在网页中的指令仅因听起来像<user>文本，即便标注为<tool>也能劫持代理。我们设计角色探测器测量模型内部对“谁在说话”的感知，发现注入文本占据与目标角色相同的表征空间。通过零样本攻击CoT Forgery，将伪造推理注入用户提示和工具输出，模型误将其当作自身思考，对前沿模型攻击成功率达60%，基线接近零。令人震惊的是，角色混淆程度可在生成任何令牌前预测攻击成功率。该机制泛化至标准代理提示注入，揭示提示注入是角色感知的可测量后果。对模型而言，听起来像某角色，就等同于真是该角色。
原文摘要 · Abstract (English)
LLMs see the world as a single stream of text, partitioned into roles like <user> or <tool>. We trace prompt injection to role confusion: models perceive the source of text from how it sounds, not its labeled role. A command hidden in a webpage hijacks an agent simply because it sounds like <user> text, despite its <tool> label. We design role probes to measure how LLMs internally perceive "who is speaking," and find that injected text occupies the same representational space as the trusted role it imitates. We demonstrate this with CoT Forgery, a zero-shot attack that injects fabricated reasoning into user prompts and tool outputs. Models mistake the forgery for their own thoughts, yielding 60% attack success against frontier models with near-zero baselines. Strikingly, the degree of role confusion predicts attack success before a single token is generated. This mechanism generalizes beyond CoT Forgery to standard agent prompt injections, revealing prompt injection as a measurable consequence of role perception. To the model, sounding like a role is indistinguishable from being one. Project page and writeup: https://role-confusion.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。