让具备深度思考能力的AI访问不可信文档,提升RAG系统安全性
Towards Safer RAG: Only Agents Capable of System 2 Thinking may Access Untrusted Documents
- 只有能进行深度思考的AI才能接触不靠谱资料
- 深度思考模型受错误信息影响小得多,准确率更高
- 适合构建更安全高效的智能问答系统
检索增强生成(RAG)显著提升了大语言模型(LLM)的表现,但其仍易受知识污染攻击——检索到的错误信息可能影响最终输出。值得注意的是,即使LLM能识别文档中的错误,仍可能被其误导。此前工作通过“隔离原则”防止答案生成模型直接访问原始证据,虽有效但带来巨大计算开销。本文提出新安全原则:仅具备反思性系统2思维能力的代理才可访问不可信文档。我们引入新指标量化错误信息识别与下游影响之间的差异,并实证对比了先进推理模型与普通模型在这些指标上的表现。结果表明,具备推理能力的模型对污染证据具有更强鲁棒性,无需严格隔离。该发现为安全RAG系统设计提供了更可行的基础。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has significantly enhanced the performance of large language models (LLMs), yet these systems remain vulnerable to knowledge-poisoning attacks, in which misinformation in retrieved documents can influence the model's final outputs. Notably, an LLM may correctly detect that a document contains incorrect information while nevertheless being influenced by it. Prior work has addressed this vulnerability through the Cordon Principle, which prevents models responsible for final answer synthesis from directly accessing raw evidence. Although effective, this strict isolation can introduce substantial computational overhead. In this work, we propose a refined security principle: only agents capable of deliberative System 2 reasoning may access untrusted documents. To evaluate this principle, we introduce novel metrics that quantify the discrepancy between misinformation detection and downstream influence. We then empirically compare state-of-the-art reasoning language models with standard language models across these metrics. Our results show that reasoning-capable models are substantially more robust to corrupted evidence, without requiring the strict isolation imposed by the Cordon Principle. These findings provide empirical support for our refined principle and suggest a more practical foundation for secure RAG system design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。