提出迭代重写方法，解决大模型多事实检索时‘中间信息丢失’问题。
FACT: Examining the Effectiveness of Iterative Context Rewriting for Multi-fact Retrieval
- 通过多轮上下文重写，逐步提取关键事实
- 在多事实检索任务中显著提升准确率
- 适合需要精准多信息提取的场景
大型语言模型在从长文本中检索单一事实方面表现良好，但在需同时获取多个事实的任务中表现不佳，尤其在生成过程中。本文揭示了一种新的‘中间信息丢失’现象：模型在生成过程中逐渐遗忘关键信息，导致检索不完整或错误。为解决该问题，我们提出一种名为FACT（Find All Crucial Texts）的迭代检索方法，通过多轮上下文重写不断优化信息捕获。该方法使模型能逐步识别并保留常被单次遍历忽略的关键事实。实验表明，FACT在多种多事实检索任务中显著提升性能，但在通用问答场景中提升有限。研究揭示了大模型在多事实检索中的局限性，强调了构建更鲁棒的长上下文检索策略的重要性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are proficient at retrieving single facts from extended contexts, yet they struggle with tasks requiring the simultaneous retrieval of multiple facts, especially during generation. This paper identifies a novel "lost-in-the-middle" phenomenon, where LLMs progressively lose track of critical information throughout the generation process, resulting in incomplete or inaccurate retrieval. To address this challenge, we introduce Find All Crucial Texts (FACT), an iterative retrieval method that refines context through successive rounds of rewriting. This approach enables models to capture essential facts incrementally, which are often overlooked in single-pass retrieval. Experiments demonstrate that FACT substantially enhances multi-fact retrieval performance across various tasks, though improvements are less notable in general-purpose QA scenarios. Our findings shed light on the limitations of LLMs in multi-fact retrieval and underscore the need for more resilient long-context retrieval strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。