用自然提问精准探测知识库文档归属，30次查询即可完成隐蔽会员推断。
Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented Generation
- 设计自然语言问题，仅当目标文档在知识库时才能回答，实现精准攻击。
- 30次查询即达成2倍于前人方法的检测率，在1%误报率下表现更优。
- 攻击隐蔽性强，检测误报率比现有方法低76倍，成本低于2美分/文档。
检索增强生成（RAG）通过外部知识库提供信息，使大语言模型生成有依据的回答，且无需修改模型参数。尽管参数未被更改避免了模型泄露，但攻击者仍可能利用上下文中检索到的文档进行成员推理。现有方法多依赖越狱或刻意构造的异常查询，易被查询重写等防护机制发现。本文提出问询攻击（Interrogation Attack, IA），针对RAG知识库中的文档设计仅在目标文档存在时可回答的自然文本问题。该方法仅需30次查询即可成功实现成员推理，且极为隐蔽：相比现有攻击，其对抗性提示被简单检测器识别的概率降低约76倍。在多种RAG配置下，该方法在1%误报率下的真正例率提升2倍，单文档推理成本不足0.02美元。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to generate grounded responses by leveraging external knowledge databases without altering model parameters. Although the absence of weight tuning prevents leakage via model parameters, it introduces the risk of inference adversaries exploiting retrieved documents in the model's context. Existing methods for membership inference and data extraction often rely on jailbreaking or carefully crafted unnatural queries, which can be easily detected or thwarted with query rewriting techniques common in RAG systems. In this work, we present Interrogation Attack (IA), a membership inference technique targeting documents in the RAG datastore. By crafting natural-text queries that are answerable only with the target document's presence, our approach demonstrates successful inference with just 30 queries while remaining stealthy; straightforward detectors identify adversarial prompts from existing methods up to ~76x more frequently than those generated by our attack. We observe a 2x improvement in TPR@1%FPR over prior inference attacks across diverse RAG configurations, all while costing less than $0.02 per document inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。