用语义叠加技术标记输入段落身份，防止提示注入攻击
Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors
- 在模型残差流中插入可学习的语义叠加层，标记输入段落属性
- 五项攻击测试中防御成功率从24.3%提升至99.0%，原始内容几乎无损
- 适合需要安全部署大模型的场景，如客服系统、代码生成工具
语言模型只能理解文本令牌。服务端虽知道每个片段的类型（用户输入、工具输出、指令等），但模型自身需维护这些信息，容易混淆或被误导：恶意文本可伪装成任意类型。提示注入正是利用此漏洞，通过混淆片段身份诱导模型执行危险操作。本文提出一种通用控制方法——语义叠加：在冻结模型的预填充位置插入小型可学习适配器，形成非文本的段落身份标注通道，无法被文本模仿。与引导向量不同，语义叠加可训练、可调节、可选择性应用。一个叠加可编码复杂语义，改变模型对指定段落的理解：当要求复制代码片段时，若该片段被标记为“另一种语言”，模型会重写为对应语言。叠加可组合、透明读取原内容，还能携带模型会执行的指令。将某段标记为“不可执行”可有效防御大量在不可信上下文中添加指令的提示注入攻击。在五个提示注入基准测试中表现优异：分离率从24.3%升至99.0%（修正了原有评分器缺陷），TensorTrust攻击成功率由34.8%降至6.2%，AlpacaFarm从99.0%降至0%，且优于所有已发表的能正常回答的防御方案，同时保持>95%字符相似度。
原文摘要 · Abstract (English)
Everything a language model sees is tokens. The serving stack knows what each span is -- user input, tool output, instructions -- but the model must keep track of that itself, and can lose track or be confused: text can be written to read like anything. Prompt injection is a natural exploit of this phenomenon. By scrambling the model's understanding of span identity, an attacker can induce unwanted and dangerous actions. Adding a non-textual channel to the model's input -- a way to communicate span identity beyond text -- mitigates this class of attack. We thus introduce a general steering technique called Semantic Overlays: small learned adapters applied at chosen prefill positions to a frozen model's residual stream. Laying an overlay over a span creates an out-of-band annotation channel that cannot be replicated by tokens. Unlike steering vectors, Semantic Overlays are trained, adaptable, and selectively applied. An overlay can encode complex semantics that reshape how the model perceives the marked span: asked to copy a code snippet under an overlay asserting a different programming language, the model rewrites the snippet in the asserted language. Overlays compose, allow transparent reading of underlying content, and can carry complex payloads -- including imperatives the model will follow. An overlay which marks a span as "non-executable" defends against the broad class of prompt injections that add instructions in untrusted context. We report strong results on five prompt injection benchmarks: SEP separation rises from 24.3% to 99.0% with utility unchanged (our scoring rule; we correct a defect in the published grader), TensorTrust attack success falls from 34.8% to 6.2%, AlpacaFarm from 99.0% to 0%, and the overlay beats every published PIArena defense that leaves the model able to answer -- while marked spans stay readable, all at >95% character similarity to the original.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。