发现并利用干扰性文本提升RAG系统准确率
The Distracting Effect: Understanding Irrelevant Passages in RAG
- 提出可量化干扰效应的新方法，识别真正有害的无关文本
- 用精心挑选的干扰文本微调模型，准确率最高提升7.5%
- 为改进RAG提供全新思路，适合做检索增强系统的研究者
RAG系统中，与查询无关的检索片段常会干扰大语言模型（LLM），导致错误回答。本文深入分析这一核心问题，提出针对查询和LLM的片段干扰效应量化方法，并验证其在不同模型间的鲁棒性。研究引入新方法，识别并利用难区分的干扰片段来优化RAG系统。通过在这些精选干扰片段上微调LLM，相比传统RAG数据集训练的模型，回答准确率最高提升7.5%。贡献在于：突破了以往将无关片段简单归为‘无关’或‘干扰’的二元分类，构建并分析多种寻找难干扰片段的方法。据我们所知，这是首个全面识别与利用难干扰片段的框架。
原文摘要 · Abstract (English)
A well-known issue with Retrieval Augmented Generation (RAG) is that retrieved passages that are irrelevant to the query sometimes distract the answer-generating LLM, causing it to provide an incorrect response. In this paper, we shed light on this core issue and formulate the distracting effect of a passage w.r.t. a query (and an LLM). We provide a quantifiable measure of the distracting effect of a passage and demonstrate its robustness across LLMs. Our research introduces novel methods for identifying and using hard distracting passages to improve RAG systems. By fine-tuning LLMs with these carefully selected distracting passages, we achieve up to a 7.5% increase in answering accuracy compared to counterparts fine-tuned on conventional RAG datasets. Our contribution is two-fold: first, we move beyond the simple binary classification of irrelevant passages as either completely unrelated vs. distracting, and second, we develop and analyze multiple methods for finding hard distracting passages. To our knowledge, no other research has provided such a comprehensive framework for identifying and utilizing hard distracting passages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。