让大模型在长文本中精准检索并推理，提升知识问答效果。
Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models
- 通过过滤噪声段落和联合训练检索头，增强模型长上下文理解能力。
- 在真实场景下测试，模型性能相比原有方法提升13至17点。
- 小模型超越GPT-4-Turbo，适合资源有限但需强推理的场景。
近期长上下文大语言模型（LCLMs）的发展为检索增强生成（RAG）带来革新，其扩展的上下文窗口使模型可直接处理完整知识库并完成检索与推理，我们称之为上下文内检索与推理（ICR²）。然而，现有基准如LOFT因上下文过于简化而高估了模型表现。为此，我们提出ICR²基准，引入由强检索器生成的干扰段落，模拟更真实场景。我们提出三种方法：(1) 先检索后生成微调，(2) 检索注意力探查法，利用注意力头在解码时过滤和去噪长上下文，(3) 与生成头联合训练检索头。在五种知名LCLM上评估显示，最佳方法应用于Mistral-7B时，在LOFT上精确匹配得分分别提升+17和+15点，在ICR²上提升+13和+2点，显著优于基线RAG和监督微调，甚至在多数任务上超越GPT-4-Turbo，尽管模型规模小得多。
原文摘要 · Abstract (English)
Recent advancements in long-context language models (LCLMs) promise to transform Retrieval-Augmented Generation (RAG) by simplifying pipelines. With their expanded context windows, LCLMs can process entire knowledge bases and perform retrieval and reasoning directly -- a capability we define as In-Context Retrieval and Reasoning (ICR^2). However, existing benchmarks like LOFT often overestimate LCLM performance by providing overly simplified contexts. To address this, we introduce ICR^2, a benchmark that evaluates LCLMs in more realistic scenarios by including confounding passages retrieved with strong retrievers. We then propose three methods to enhance LCLM performance: (1) retrieve-then-generate fine-tuning, (2) retrieval-attention-probing, which uses attention heads to filter and de-noise long contexts during decoding, and (3) joint retrieval head training alongside the generation head. Our evaluation of five well-known LCLMs on LOFT and ICR^2 demonstrates significant gains with our best approach applied to Mistral-7B: +17 and +15 points by Exact Match on LOFT, and +13 and +2 points on ICR^2, compared to vanilla RAG and supervised fine-tuning, respectively. It even outperforms GPT-4-Turbo on most tasks despite being a much smaller model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。