通过分析注意力模式，检测大模型中的指令注入攻击。
Attention Tracker: Detecting Prompt Injection Attacks in LLMs
- 发现关键注意力头会转移焦点至恶意指令，形成干扰效应。
- 无需额外推理，检测准确率比现有方法最高提升10.0% AUROC。
- 适用于各类模型和攻击类型，尤其适合小模型部署。
大型语言模型（LLMs）虽在多个领域带来变革，但仍易受指令注入攻击影响，即恶意输入诱使模型忽略原始指令并执行指定操作。本文通过分析LLM内部的注意力模式，揭示此类攻击的机制：特定注意力头（称作重要头）会将关注点从原指令转移到注入指令，形成‘干扰效应’。基于此，我们提出无需训练的检测方法Attention Tracker，通过追踪指令相关的注意力分布来识别攻击，无需额外的LLM推理。该方法在多种模型、数据集和攻击类型下均表现良好，相比现有方法最高提升10.0% AUROC，且在小型模型上仍具有效性。通过广泛评估验证了其鲁棒性，并为防范集成式LLM系统中的指令注入漏洞提供了新思路。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have revolutionized various domains but remain vulnerable to prompt injection attacks, where malicious inputs manipulate the model into ignoring original instructions and executing designated action. In this paper, we investigate the underlying mechanisms of these attacks by analyzing the attention patterns within LLMs. We introduce the concept of the distraction effect, where specific attention heads, termed important heads, shift focus from the original instruction to the injected instruction. Building on this discovery, we propose Attention Tracker, a training-free detection method that tracks attention patterns on instruction to detect prompt injection attacks without the need for additional LLM inference. Our method generalizes effectively across diverse models, datasets, and attack types, showing an AUROC improvement of up to 10.0% over existing methods, and performs well even on small LLMs. We demonstrate the robustness of our approach through extensive evaluations and provide insights into safeguarding LLM-integrated systems from prompt injection vulnerabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。