揭示RAG系统知识泄露风险，提出可配置评估框架
LeakDojo: Decoding the Leakage Threats of RAG Systems
- 构建可配置框架LeakDojo，系统化评估RAG泄露风险
- 发现指令遵循能力越强，泄露风险越高；查询生成与恶意指令独立贡献泄露
- 提升RAG准确性可能反而增加泄露风险，适合安全研究人员参考
检索增强生成（RAG）使大语言模型（LLMs）能利用外部知识，但也暴露了其数据库面临泄露攻击的风险。随着RAG系统复杂度上升及LLM指令遵循能力增强，现有研究难以系统评估此类风险。本文提出LeakDojo，一个可配置的可控评估框架。基于该框架，我们在14个LLM、4个数据集和多种RAG系统上对6种现有攻击进行了基准测试。研究发现：(1) 查询生成与对抗性指令对泄露有独立贡献，整体泄露风险可近似为两者的乘积；(2) 指令遵循能力越强，泄露风险越高；(3) 提升RAG忠实度可能引入更高的泄露风险。这些发现为实际中理解和缓解RAG泄露提供了切实可行的指导。代码已开源。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) enables large language models (LLMs) to leverage external knowledge, but also exposes valuable RAG databases to leakage attacks. As RAG systems grow more complex and LLMs exhibit stronger instruction-following capabilities, existing studies fall short of systematically assessing RAG leakage risks. We present LeakDojo, a configurable framework for controlled evaluation of RAG leakage. Using LeakDojo, we benchmark six existing attacks across fourteen LLMs, four datasets, and diverse RAG systems. Our study reveals that (1) query generation and adversarial instructions contribute independently to leakage, with overall leakage well approximated by their product; (2) stronger instruction-following capability correlates with higher leakage risk; and (3) improvements in RAG faithfulness can introduce increased leakage risk. These findings provide actionable insights for understanding and mitigating RAG leakage in practice. Our codebase is available at https://github.com/yeasen-z/LeakDojo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。