arXiv:2509.03787cs.IRcs.CL2025-09被引 9

测试医疗领域RAG系统对恶意证据的鲁棒性,发现引入有用信息可提升安全性。

Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domain

  • 通过控制检索文档类型(有益、有害、恶意)评估RAG在医疗问答中的表现。
  • 恶意文档使模型输出与真实答案偏差显著增大,准确率下降超40%。
  • 当检索池中同时存在有益信息时,系统鲁棒性明显增强,适合高风险场景研究者参考。

检索增强生成(RAG)系统通过提供检索到的证据来约束大语言模型(LLM)的回答,以减少幻觉并提升其在训练数据范围外的问答准确性。然而,该设计存在关键漏洞:若检索到的证据包含错误信息,模型可能吸收并复现这些误导内容。这一问题在存在专门制造的恶意证据时尤为严重。本文针对医疗领域开展系统性评估,分析模型输出与真实答案的一致性。鉴于医疗错误回答可能造成严重危害,且多数健康问题具备基于证据的真实答案,本研究选取常见健康问题进行控制实验,改变检索文档类型(有益、有害、恶意)及用户提问方式(一致、中立、不一致)。结果表明,恶意文档会显著降低输出对齐度,但若检索池中同时包含有益证据,系统鲁棒性仍可保持。研究为高风险领域安全RAG系统的设计提供了可行方案,并公开所有实验数据以支持后续研究。

原文摘要 · Abstract (English)

Retrieval augmented generation (RAG) systems provide a method for factually grounding the responses of a Large Language Model (LLM) by providing retrieved evidence, or context, as support. Guided by this context, RAG systems can reduce hallucinations and expand the ability of LLMs to accurately answer questions outside the scope of their training data. Unfortunately, this design introduces a critical vulnerability: LLMs may absorb and reproduce misinformation present in retrieved evidence. This problem is magnified if retrieved evidence contains adversarial material explicitly intended to promulgate misinformation. This paper presents a systematic evaluation of RAG robustness in the health domain and examines alignment between model outputs and ground-truth answers. We focus on the health domain due to the potential for harm caused by incorrect responses, as well as the availability of evidence-based ground truth for many common health-related questions. We conduct controlled experiments using common health questions, varying both the type and composition of the retrieved documents (helpful, harmful, and adversarial) as well as the framing of the question by the user (consistent, neutral, and inconsistent). Our findings reveal that adversarial documents substantially degrade alignment, but robustness can be preserved when helpful evidence is also present in the retrieval pool. These findings offer actionable insights for designing safer RAG systems in high-stakes domains by highlighting the need for retrieval safeguards. To enable reproducibility and facilitate future research, all experimental results are publicly available in our github repository. https://github.com/shakibaam/RAG_ROBUSTNESS_EVAL

RAG医疗AI对抗攻击可靠性评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。