通过检测嵌入空间语义漂移,零样本识别大模型中的提示注入攻击。
Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMs
- 利用干净与可疑输入的嵌入差异,通过余弦相似度捕捉语义漂移。
- 在多个模型上实现超93%检测准确率,误报率低于3%。
- 无需训练或模型内部信息,适合快速部署到各类大模型系统。
提示注入攻击已成为大模型应用的严重漏洞,攻击者通过邮件或用户生成内容等间接输入通道绕过对齐机制,诱导有害输出。尽管对齐技术不断进步,现有大模型仍普遍易受攻击,亟需高效、通用且可扩展的检测方法。本文提出零样本嵌入漂移检测(ZEDD),一种轻量级、低工程成本的框架,通过量化正常输入与可疑输入在嵌入空间中的语义偏移,识别直接与间接提示注入。ZEDD无需访问模型内部结构、攻击先验知识或任务微调,可零样本部署于多种大模型架构。方法基于对抗性-清洁提示对,以余弦相似度衡量嵌入漂移,捕捉真实攻击中的细微操纵。我们构建并重标注了涵盖五类攻击的大型数据集LLMail-Inject,源自公开来源。大量实验表明,嵌入漂移是稳健且可迁移的信号,在Llama 3、Qwen 2、Mistral等模型上检测准确率超93%,误报率低于3%,显著优于传统方法,为现有大模型流水线提供可集成的轻量级安全防御层。
原文摘要 · Abstract (English)
Prompt injection attacks have become an increasing vulnerability for LLM applications, where adversarial prompts exploit indirect input channels such as emails or user-generated content to circumvent alignment safeguards and induce harmful or unintended outputs. Despite advances in alignment, even state-of-the-art LLMs remain broadly vulnerable to adversarial prompts, underscoring the urgent need for robust, productive, and generalizable detection mechanisms beyond inefficient, model-specific patches. In this work, we propose Zero-Shot Embedding Drift Detection (ZEDD), a lightweight, low-engineering-overhead framework that identifies both direct and indirect prompt injection attempts by quantifying semantic shifts in embedding space between benign and suspect inputs. ZEDD operates without requiring access to model internals, prior knowledge of attack types, or task-specific retraining, enabling efficient zero-shot deployment across diverse LLM architectures. Our method uses adversarial-clean prompt pairs and measures embedding drift via cosine similarity to capture subtle adversarial manipulations inherent to real-world injection attacks. To ensure robust evaluation, we assemble and re-annotate the comprehensive LLMail-Inject dataset spanning five injection categories derived from publicly available sources. Extensive experiments demonstrate that embedding drift is a robust and transferable signal, outperforming traditional methods in detection accuracy and operational efficiency. With greater than 93% accuracy in classifying prompt injections across model architectures like Llama 3, Qwen 2, and Mistral and a false positive rate of <3%, our approach offers a lightweight, scalable defense layer that integrates into existing LLM pipelines, addressing a critical gap in securing LLM-powered systems to withstand adaptive adversarial threats.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。