提出评估代理检测生成式AI中的虚假信息与知识污染，提升RAG系统可信度。
Trustworthy RAG: An Evaluation Agent for Detecting Misinformation and Knowledge Poisoning in Generative AI Systems
- 用自然语言推理与五信号检测结合，验证检索内容真实性。
- 在TruthfulQA上达到91%准确率，指令注入检测100%召回率。
- 适合需要高可靠性的安全编码、医疗等关键场景使用。
检索增强生成（RAG）将大语言模型输出锚定于外部知识，但通常盲目信任检索结果，造成安全-可靠性差距：语义相关不等于事实正确。攻击者通过知识污染插入恶意文档，引发定向误导。我们提出评估代理，作为中间件融合自然语言推断事实验证、五信号污染检测及加权聚合，并设计信任指数 T = 0.4F + 0.35C + 0.25(1-P)，在高污染场景引入非线性衰减器。在 TruthfulQA 数据集上，基于 Llama 3.3 70B 的系统达到 91% 准确率和 100% 精确率，指令注入检测实现 100% 召回率；而实体替换等在位编辑仍难检测。跨三种 LLM，信任指数保持判别性，受试者工作特征曲线下面积（ROC-AUC）为 0.73 至 0.81；生成风格比模型规模影响更大，每模型阈值校准可恢复基线竞争力；较弱的 FEVER 结果表明跨数据集泛化需领域特定校准。在软件工程场景中，基于 OWASP Top 10 和 CWE 的安全编码助手，该代理成功阻断不安全建议的指令注入（F1 92%），但矛盾和细微语义削弱仍难识别。整个系统衡量的是污染上下文的检测能力，而非模型是否采纳误导信息。代码、攻击生成器与实验数据已开源。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) grounds Large Language Model (LLM) outputs in external knowledge, but RAG systems usually trust whatever they retrieve, creating a Security-Reliability Gap: high semantic relevance does not guarantee factual truth. Adversaries exploit this through knowledge poisoning, inserting malicious documents to cause targeted misinformation. We propose an Evaluation Agent, middleware that combines Natural Language Inference (NLI) factual verification, a five-signal poison detector with relevance-weighted aggregation, and a Trust Index T = 0.4 F + 0.35 C + 0.25 (1 - P ) with a non-linear dampener for high-contamination contexts. On TruthfulQA with Llama 3.3 70B, the agent reaches 91% accuracy and 100% precision, with 100% recall on instruction injection, while in-place edits, such as entity swaps, remain hard to detect. Across three LLMs the Trust Index stays discriminative, with a Receiver Operating Characteristic Area Under the Curve (ROC-AUC) of 0.73 to 0.81; generation style matters more than model size, and per-LLM threshold calibration restores baseline competitive accuracy, whereas a weaker FEVER result shows that cross-dataset generalization requires domain-specific calibration. In a software-engineering use case, a secure-coding assistant over guidance from the Open Worldwide Application Security Project (OWASP) Top 10 and the Common Weakness Enumeration (CWE), the agent reliably blocks instruction injection of unsafe advice (F1 92%), while contradiction and subtle semantic weakening remain hard. Throughout, the agent measures detection of poisoned context before generation, not whether the LLM adopts the injected misinformation. We release the proposed approach, attack generator, and experimental artifacts at the link: https://github.com/GPT-Laboratory/TrustworthyRAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。