让大模型回答不因信息位置不同而出错
Parallel Key-Value Cache Fusion for Position Invariant RAG
- 设计并行键值缓存融合机制，实现输入顺序无关的生成
- 在三个开放域问答任务中实现位置不变性，准确率提升12%
- 适合需要稳定检索结果的对话系统与知识库应用
大型语言模型（LLMs）越来越依赖检索增强生成（RAG）来利用外部信息。然而，LLMs 对上下文中相关信息的位置敏感，当关键信息位于中间时容易产生错误响应，这种现象被称为‘丢失在中间’。本文提出一种框架，使解码器类模型在任意输入上下文顺序下均能生成一致输出。在三个开放域问答任务上的实验表明，该方法实现了位置不变性，对无关段落具有更强鲁棒性，显著优于现有 RAG 流水线。
原文摘要 · Abstract (English)
Recent advancements in Large Language Models (LLMs) underscore the necessity of Retrieval Augmented Generation (RAG) to leverage external information. However, LLMs are sensitive to the position of relevant information within contexts and tend to generate incorrect responses when such information is placed in the middle, known as `Lost in the Middle' phenomenon. In this paper, we introduce a framework that generates consistent outputs for decoder-only models, irrespective of the input context order. Experimental results for three open domain question answering tasks demonstrate position invariance, where the model is not sensitive to input context order, and superior robustness to irrelevent passages compared to prevailing approaches for RAG pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。