提出动态门控机制,让模型自动判断是否需要外部信息。
Context Awareness Gate For Retrieval Augmented Generation
- 设计上下文感知门控,根据查询需求决定是否引入外部检索内容。
- 提出向量候选方法,统计独立且可扩展,提升检索相关性。
- 分析上下文与问题关系分布,为RAG系统优化提供数据支持。
检索增强生成(RAG)已成为缓解大语言模型在回答领域特定问题时局限性的广泛采用方法。以往研究主要关注提升检索数据块的准确性和质量以增强生成流程整体性能。然而,尽管持续进步,检索无关信息这一关键问题——会削弱模型有效利用内部知识的能力——却未得到足够重视。本文研究了在开放域问答中检索无关信息的影响,揭示其对大语言模型输出质量的显著负面影响。为此,我们提出上下文感知门控(CAG)架构,一种新型机制,可动态调整大语言模型的输入提示,依据用户查询是否需要外部上下文检索。此外,我们引入向量候选方法,作为CAG的核心数学组件,具备统计性、不依赖于大语言模型且高度可扩展。我们还进一步分析了上下文与问题间关系的分布,该分析可用于改进检索增强生成(RAG)系统的上下文检索过程。
原文摘要 · Abstract (English)
Retrieval Augmented Generation (RAG) has emerged as a widely adopted approach to mitigate the limitations of large language models (LLMs) in answering domain-specific questions. Previous research has predominantly focused on improving the accuracy and quality of retrieved data chunks to enhance the overall performance of the generation pipeline. However, despite ongoing advancements, the critical issue of retrieving irrelevant information -- which can impair the ability of the model to utilize its internal knowledge effectively -- has received minimal attention. In this work, we investigate the impact of retrieving irrelevant information in open-domain question answering, highlighting its significant detrimental effect on the quality of LLM outputs. To address this challenge, we propose the Context Awareness Gate (CAG) architecture, a novel mechanism that dynamically adjusts the LLMs' input prompt based on whether the user query necessitates external context retrieval. Additionally, we introduce the Vector Candidates method, a core mathematical component of CAG that is statistical, LLM-independent, and highly scalable. We further examine the distributions of relationships between contexts and questions, presenting a statistical analysis of these distributions. This analysis can be leveraged to enhance the context retrieval process in Retrieval Augmented Generation (RAG) systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。