通过增强中间表示提升指令层级防御力，显著降低提示注入攻击成功率。
Stronger Enforcement of Instruction Hierarchy via Augmented Intermediate Representations
- 在模型中间层注入可训练的特权嵌入，动态标识指令层级。
- 对梯度攻击的防御效果提升1.6至9.2倍，且不影响模型性能。
- 适合关注LLM安全、对抗攻击防御的研究者与工程师。
提示注入攻击是大语言模型（LLMs）的关键安全漏洞，攻击者可通过输入上下文中嵌入恶意指令来劫持模型行为。现有防御机制依赖指令层级（IH）信号，通常通过特殊分隔符或加性嵌入在初始输入层标注令牌权限级别。然而，此类方法仅在输入层注入信号，我们推测这限制了其在模型深层传播中区分令牌权限的能力。为此，我们提出一种新方法：将IH信号注入网络中的中间令牌表示。通过在各层添加编码权限信息的可训练嵌入，增强中间表示。在多个模型和训练方法上的评估表明，本方案在梯度基提示注入攻击中，攻击成功率相较最先进方法降低1.6至9.2倍，且未明显损害模型可用性。
原文摘要 · Abstract (English)
Prompt injection attacks are a critical security vulnerability in large language models (LLMs), allowing attackers to hijack model behavior by injecting malicious instructions within the input context. Recent defense mechanisms have leveraged an Instruction Hierarchy (IH) Signal, often implemented through special delimiter tokens or additive embeddings to denote the privilege level of input tokens. However, these prior works typically inject the IH signal exclusively at the initial input layer, which we hypothesize limits its ability to effectively distinguish the privilege levels of tokens as it propagates through the different layers of the model. To overcome this limitation, we introduce a novel approach that injects the IH signal into the intermediate token representations within the network. Our method augments these representations with layer-specific trainable embeddings that encode the privilege information. Our evaluations across multiple models and training methods reveal that our proposal yields between $1.6\times$ and $9.2\times$ reduction in attack success rate on gradient-based prompt injection attacks compared to state-of-the-art methods, without significantly degrading the model's utility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。