通过检测模型行为状态，识别并防御外部数据中的隐蔽指令攻击
Defending against Indirect Prompt Injection by Instruction Detection
- 利用模型中间层隐藏状态与梯度特征检测嵌入指令
- 在域内/域外场景下检测准确率达99.6%/96.9%，攻击成功率降至0.03%
- 适用于RAG系统等集成外部数据的LLM应用
大型语言模型（LLM）与外部数据源的融合日益普遍，检索增强生成（RAG）是典型代表。然而，这种整合带来了间接提示注入（IPI）攻击漏洞，即外部数据中嵌入的隐藏指令可诱导模型执行非预期或有害行为。我们发现，IPI攻击依赖于外部内容中指令的存在，这些指令会改变模型的行为状态。能否通过检测此类状态变化来防御攻击？本文提出InstructDetector，一种基于行为状态检测的新方法。具体而言，我们证明中间层的隐藏状态和梯度具有高度区分性，可用于指令识别。通过有效融合这些特征，InstructDetector在域内设置下检测准确率达99.60%，域外设置下达96.90%，并在BIPIA基准上将攻击成功率降至0.03%。代码已公开于https://github.com/MYVAE/Instruction-detection。
原文摘要 · Abstract (English)
The integration of Large Language Models (LLMs) with external sources is becoming increasingly common, with Retrieval-Augmented Generation (RAG) being a prominent example. However, this integration introduces vulnerabilities of Indirect Prompt Injection (IPI) attacks, where hidden instructions embedded in external data can manipulate LLMs into executing unintended or harmful actions. We recognize that IPI attacks fundamentally rely on the presence of instructions embedded within external content, which can alter the behavioral states of LLMs. Can the effective detection of such state changes help us defend against IPI attacks? In this paper, we propose InstructDetector, a novel detection-based approach that leverages the behavioral states of LLMs to identify potential IPI attacks. Specifically, we demonstrate the hidden states and gradients from intermediate layers provide highly discriminative features for instruction detection. By effectively combining these features, InstructDetector achieves a detection accuracy of 99.60% in the in-domain setting and 96.90% in the out-of-domain setting, and reduces the attack success rate to just 0.03% on the BIPIA benchmark. The code is publicly available at https://github.com/MYVAE/Instruction-detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。