通过分析大模型激活模式，可精准识别检索增强生成中的恶意攻击。
RevPRAG: Revealing Poisoning Attacks in Retrieval-Augmented Generation through LLM Activation Analysis
- 利用大模型激活特征差异区分正常与中毒响应
- 在多个数据集上实现98%检出率，误报率接近1%
- 适合关注RAG系统安全性的研究人员和开发者
检索增强生成（RAG）通过从知识库中检索相关信息来丰富大模型输入，提升生成内容的准确性和上下文相关性。然而，由于知识库来自公开渠道（如维基百科），可能引入新的攻击面。RAG poisoning攻击通过向知识库注入恶意文本，诱导模型生成攻击者指定的中毒响应。目前缺乏有效的检测手段。本文提出RevPRAG，一种灵活且自动化的检测流程，利用大模型激活特征识别中毒响应。研究发现，生成正确响应与中毒响应时，模型激活模式存在显著差异。在多个基准数据集和RAG架构上，该方法实现98%真阳性率，同时保持接近1%的假阳性率。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) enriches the input to LLMs by retrieving information from the relevant knowledge database, enabling them to produce responses that are more accurate and contextually appropriate. It is worth noting that the knowledge database, being sourced from publicly available channels such as Wikipedia, inevitably introduces a new attack surface. RAG poisoning involves injecting malicious texts into the knowledge database, ultimately leading to the generation of the attacker's target response (also called poisoned response). However, there are currently limited methods available for detecting such poisoning attacks. We aim to bridge the gap in this work. Particularly, we introduce RevPRAG, a flexible and automated detection pipeline that leverages the activations of LLMs for poisoned response detection. Our investigation uncovers distinct patterns in LLMs' activations when generating correct responses versus poisoned responses. Our results on multiple benchmark datasets and RAG architectures show our approach could achieve 98% true positive rate, while maintaining false positive rates close to 1%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。