提升企业问答系统可解释性,让答案来源更清晰、因果更可信。
Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems
- 用元数据和上下文文本增强检索证据的语境信息
- 通过移除证据对比回答变化,实现因果层面的归因分析
- 适用于需要高可信度解释的企业级对话问答场景
检索增强生成(RAG)是企业私有数据对话式问答(ConvQA)的核心架构。现有RAG系统存在两大缺陷:一是检索结果仅含原始文本,缺乏文档上下文,影响检索与回答质量;二是归因方法仅依赖答案与检索片段的相似性,生成的是合理而非因果解释。本文提出RAGONITE系统,通过(i)利用源元数据和周围文本对证据进行上下文化;(ii)引入反事实归因,通过移除某证据后回答的变化来评估其贡献,实现因果解释。为验证方法,我们发布新基准ConfQuestions:包含300个手工创建的中英文对话问题,对应215个公开Confluence页面的完整问题-答案对及真实链接。实验表明,上下文化显著提升RAG性能,反事实归因优于传统归因方式。
原文摘要 · Abstract (English)
Retrieval Augmented Generation (RAG) works as a backbone for interacting with an enterprise's own data via Conversational Question Answering (ConvQA). In a RAG system, a retriever fetches passages from a collection in response to a question, which are then included in the prompt of a large language model (LLM) for generating a natural language (NL) answer. However, several RAG systems today suffer from two shortcomings: (i) retrieved passages usually contain their raw text and lack appropriate document context, negatively impacting both retrieval and answering quality; and (ii) attribution strategies that explain answer generation typically rely only on similarity between the answer and the retrieved passages, thereby only generating plausible but not causal explanations. In this work, we demonstrate RAGONITE, a RAG system that remedies the above concerns by: (i) contextualizing evidence with source metadata and surrounding text; and (ii) computing counterfactual attribution, a causal explanation approach where the contribution of an evidence to an answer is determined by the similarity of the original response to the answer obtained by removing that evidence. To evaluate our proposals, we release a new benchmark ConfQuestions: it has 300 hand-created conversational questions, each in English and German, coupled with ground truth URLs, completed questions, and answers from 215 public Confluence pages. These documents are typical of enterprise wiki spaces with heterogeneous elements. Experiments with RAGONITE on ConfQuestions show the viability of our ideas: contextualization improves RAG performance, and counterfactual explanations outperform standard attribution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。