arXiv:2601.05260cs.IRcs.AI2026-01

提出新指标衡量文档对大模型输出的影响,提升RAG系统可信度。

Quantifying Document Impact in RAG-LLMs

  • 基于部分信息分解设计影响得分,量化每篇检索文档的贡献。
  • 在毒化攻击实验中,86%情况下正确识别出恶意文档为最高影响项。
  • 仅用影响得分高的文档生成的内容更接近原始输出,证明其有效性。

检索增强生成(RAG)通过连接外部知识提升大语言模型的准确性并减少过时信息,但也带来事实不一致、来源冲突、偏见传播和安全漏洞等问题,削弱了系统的可信性。当前RAG评估的一大空白是缺乏量化单篇检索文档对最终输出贡献的指标。为此,我们提出影响得分(IS),一种基于部分信息分解的新指标,用于测量每篇检索文档对生成结果的影响。通过两个实验验证:首先,在三个数据集上进行毒化攻击模拟,结果显示在86%的情况下IS正确识别出恶意文档为最具影响力项;其次,消融实验表明,仅使用IS排名靠前的文档生成的响应,其与原始响应的相似度始终高于仅使用其余文档生成的结果。这些结果证实了IS在隔离和量化文档影响方面的有效性,为提升RAG系统的透明性与可靠性提供了有力工具。

原文摘要 · Abstract (English)

Retrieval Augmented Generation (RAG) enhances Large Language Models (LLMs) by connecting them to external knowledge, improving accuracy and reducing outdated information. However, this introduces challenges such as factual inconsistencies, source conflicts, bias propagation, and security vulnerabilities, which undermine the trustworthiness of RAG systems. A key gap in current RAG evaluation is the lack of a metric to quantify the contribution of individual retrieved documents to the final output. To address this, we introduce the Influence Score (IS), a novel metric based on Partial Information Decomposition that measures the impact of each retrieved document on the generated response. We validate IS through two experiments. First, a poison attack simulation across three datasets demonstrates that IS correctly identifies the malicious document as the most influential in $86\%$ of cases. Second, an ablation study shows that a response generated using only the top-ranked documents by IS is consistently judged more similar to the original response than one generated from the remaining documents. These results confirm the efficacy of IS in isolating and quantifying document influence, offering a valuable tool for improving the transparency and reliability of RAG systems.

RAG评估指标可信度文档影响

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。