提出新方法检测知识库中的恶意文档，保护大模型生成安全。
Safeguarding RAG Pipelines with GMTP: A Gradient-based Masked Token Probability Method for Poisoned Document Detection
- 通过梯度分析找出关键影响词汇，再用掩码模型检测其概率异常。
- 在多种攻击下可过滤超90%恶意内容，同时保留有效信息。
- 适合关注大模型安全、RAG系统防御的研究者与开发者。
检索增强生成（RAG）通过引入外部知识提升大语言模型的准确性与时效性，但其依赖外部数据源也带来安全风险：攻击者可向知识库注入毒化文档，诱导模型生成有害或误导性输出。本文提出基于梯度的掩码词概率方法（GMTP），用于检测并过滤恶意文档。具体而言，GMTP通过分析检索器相似度函数的梯度，识别对结果影响大的关键词汇；随后对这些词汇进行掩码，并利用掩码语言模型（MLM）评估其被正确恢复的概率。由于注入的恶意词汇通常具有显著偏低的掩码概率，因此可被高效识别。实验表明，GMTP能在多种数据集和对抗场景下过滤超过90%的毒化内容，同时保持相关文档的可用性，确保检索与生成性能稳定可靠。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by providing external knowledge for accurate and up-to-date responses. However, this reliance on external sources exposes a security risk, attackers can inject poisoned documents into the knowledge base to steer the generation process toward harmful or misleading outputs. In this paper, we propose Gradient-based Masked Token Probability (GMTP), a novel defense method to detect and filter out adversarially crafted documents. Specifically, GMTP identifies high-impact tokens by examining gradients of the retriever's similarity function. These key tokens are then masked, and their probabilities are checked via a Masked Language Model (MLM). Since injected tokens typically exhibit markedly low masked-token probabilities, this enables GMTP to easily detect malicious documents and achieve high-precision filtering. Experiments demonstrate that GMTP is able to eliminate over 90% of poisoned content while retaining relevant documents, thus maintaining robust retrieval and generation performance across diverse datasets and adversarial settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。