通过遮蔽关键词检测大模型知识库是否包含特定文档,提升隐私泄露风险识别能力。
Mask-based Membership Inference Attacks for Retrieval-Augmented Generation
- 用遮蔽算法对目标文档部分词进行掩码,生成提示输入RAG系统
- 若文档在知识库中,系统可准确还原被遮蔽内容,预测准确率显著更高
- 方法更专注文档本身,不受其他文本或模型内知识干扰,适合数据版权保护场景
检索增强生成(RAG)通过引入实时、领域相关的知识,有效缓解大语言模型的幻觉问题。近年来,越来越多的实时或受版权保护的数据被存入RAG知识库而非用于模型训练,这引发了成员推断攻击(MIAs)的担忧——即判断特定文档是否存在于RAG系统的知识库中,以保护数据生产者的权利。现有RAG系统中的成员推断攻击仍不充分:部分方法仅依赖RAG系统的判断,易受其他文档或模型内部知识影响,不可靠且缺乏可解释性。为此,本文提出一种基于遮蔽的成员推断攻击(MBA)框架。该框架首先使用遮蔽算法对目标文档中的部分词汇进行掩码,再将掩码后文本作为提示输入RAG系统,要求其预测被遮蔽内容。若目标文档存在于知识库中,系统可检索到完整原文作为上下文,实现高精度预测。最后采用简单有效的阈值法,根据预测准确率推断文档是否属于知识库。该方法更具文档特异性,使生成结果更少受其他文档或模型内知识干扰。大量实验表明,相比现有基线模型,本方法在成员推断任务上表现更优。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has been an effective approach to mitigate hallucinations in large language models (LLMs) by incorporating up-to-date and domain-specific knowledge. Recently, there has been a trend of storing up-to-date or copyrighted data in RAG knowledge databases instead of using it for LLM training. This practice has raised concerns about Membership Inference Attacks (MIAs), which aim to detect if a specific target document is stored in the RAG system's knowledge database so as to protect the rights of data producers. While research has focused on enhancing the trustworthiness of RAG systems, existing MIAs for RAG systems remain largely insufficient. Previous work either relies solely on the RAG system's judgment or is easily influenced by other documents or the LLM's internal knowledge, which is unreliable and lacks explainability. To address these limitations, we propose a Mask-Based Membership Inference Attacks (MBA) framework. Our framework first employs a masking algorithm that effectively masks a certain number of words in the target document. The masked text is then used to prompt the RAG system, and the RAG system is required to predict the mask values. If the target document appears in the knowledge database, the masked text will retrieve the complete target document as context, allowing for accurate mask prediction. Finally, we adopt a simple yet effective threshold-based method to infer the membership of target document by analyzing the accuracy of mask prediction. Our mask-based approach is more document-specific, making the RAG system's generation less susceptible to distractions from other documents or the LLM's internal knowledge. Extensive experiments demonstrate the effectiveness of our approach compared to existing baseline models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。