用考试题形式检测RAG系统是否收录了某文档,隐蔽且精准。
E-MIA: Exam-Style Black-Box Membership Inference Attacks against RAG Systems

- 将文档中的硬证据转为四类客观评分的考题,用总分判断是否被收录。
- 在多个数据集上实现高区分度,成员与非成员得分分布明显分离。
- 攻击隐蔽不触发防御,适合研究隐私泄露或评估RAG安全性的人员。
检索增强生成(RAG)通过推理时检索外部文档提升大模型能力,但也使检索语料库成为敏感资产。在黑盒场景下,攻击者仅通过查询响应交互,即可推断候选文档是否被纳入RAG知识库(即文档级成员推断),从而泄露语料覆盖范围和敏感话题存在性。现有RAG成员推断方法要么依赖语义相似度等软信号,导致成员与非成员得分分布重叠、阈值不稳定;要么使用显式确认探针,意图明显易被拒绝或检测。本文提出E-MIA,将目标文档中的可验证硬证据(如细粒度细节、专有名词/术语、定义性陈述、元数据线索及因果/约束关系)转化为四类客观评分题目(填空/单选/多选/判断),并以多题综合得分作为成员信号。在多个数据集和多样RAG配置下的实验表明,E-MIA在严格条件下显著提升成员与非成员的可区分性,同时保持自然、隐蔽的查询行为。我们进一步分析了题目构成与试卷长度对攻击效果的影响。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) equips large language models (LLMs) with external evidence by retrieving documents at inference time, but it also turns the retrieval corpusinto a sensitive asset. Under a black-box setting, an adversary given a candidate document can infer whether it has been ingested into the RAG knowledge base (i.e., document-level membership inference) solely from query response interactions, thereby leaking corpus coverage and the existence of sensitive topics. Existing RAG MIA methods either rely on soft signals such as semantic similarity, which often yield overlapping member/non-member score distributions and unstable thresholds, or employ explicit confirmation probes whose intent is conspicuous and thus prone to refusal and detection. We propose E-MIA, which converts verifiable hard evidence in the target document (e.g., fine-grained details, proper nouns/technical terms, definitional statements, metadata cues, and causal/constraint relations) into an exam with four objectively gradable question types (FB/SC/MC/T/F), and uses the aggregated exam score across multiple evidence targeted questions as the membership signal. Experiments across multiple datasets and diverse RAG configurations demonstrate that E-MIA improves member/non-member separability in stringent settings while preserving natural, stealthy queries, and we further analyze the impact of question composition and exam length on attack effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。