提出分层一致性框架，可检出RAG系统中隐藏的矛盾证据。
A Hierarchical Consistency Framework for Auditing Retrieval-Augmented Generation Systems
- 分三级审计RAG：知识库、检索上下文、生成答案
- 在100个案例中发现答案正确但上下文自相矛盾
- 适合评估RAG系统可靠性，尤其对安全敏感场景
检索增强生成（RAG）通常仅通过最终答案是否正确来评估，但这种测试不足：答案可能匹配参考答案，而其生成所依据的上下文却存在直接矛盾，这些争议证据在仅看答案的审查中无法察觉，且检索相关性评分也无法发现。本文提出分层一致性框架（HCF），一种后置、模型无关的审计方法，用于检查RAG过程的三个不同层级：知识语料库、最终检索到的上下文、生成的答案。HCF将语料库中的冲突表示为带来源链接的原子事实，从而识别出责任文档，并为每个答案提供答案一致性得分（ACS）及支持与矛盾陈述的解释。我们在涵盖五个领域、100个查询-语料实例的若干受控语料库上评估了HCF。人工评估者将每个生成响应与其提供的真实响应进行比较。结果表明，三个诊断层级可分离：检索相似度均值最高的语料库，其平均ACS最低；结构退化的语料库在语料库层级表现更差，但在答案层级表现更好。最重要的是，HCF在多个案例中识别出与答案一致但存在矛盾的检索证据，尽管答案仍匹配真实答案。HCF不验证事实真伪，而是使支持和挑战答案的证据变得可审查且可追溯。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) is commonly evaluated by whether the final answer is correct. That test is insufficient: an answer can match its reference while the context that produced it contains a direct contradiction, leaving the contested evidence invisible to answer-only review and retrieval relevance scores. This paper presents the Hierarchical Consistency Framework (HCF), a post-hoc, model-agnostic audit of three distinct levels of a RAG process: the knowledge corpus, the final retrieved context, and the generated answer. HCF represents corpus conflicts as source-linked atomic facts, thereby identifying the documents responsible, and returns each Answer Consistency Score (ACS) with an explanation of supporting and contradictory contextual statements. We evaluate HCF on several controlled corpora spanning five domains and 100 query-corpus instances. A human evaluator compares every generated response with its supplied ground-truth response. The results show that the three diagnostic levels can dissociate: the corpus with the highest mean retrieval similarity has the lowest mean ACS, while a structurally degraded corpus performs worse at corpus level but better at answer level. Most importantly, HCF identifies contradictory retrieved evidence in several cases where the answer still matches the ground truth. HCF does not certify factual truth; it makes the evidence supporting and challenging an answer inspectable and attributable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。