通过两阶段筛选提升长文档问答准确率
CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capability
- 分粗细两阶段，用检索头逐步过滤无关文档
- 在Mistral模型上比SFT和RAG分别提升22.1%和13.7%的SubEM
- 适合需要精准长文档推理的场景
大型语言模型（LLMs）虽已扩展输入上下文长度，但在长文本中的检索与推理能力仍有限。现有方法虽采用提示策略和检索头缓解此问题，但难以平衡召回率与精确率。为此，我们提出CAFE，一种基于检索头的粗到精两阶段信息搜索方法，以增强多文档问答能力。首先通过粗粒度过滤利用检索头识别并排序相关文档；随后细粒度引导机制聚焦最相关的内容。实验表明，CAFE在多个基准测试中优于基线方法，在Mistral模型上分别较SFT和RAG提升22.1%和13.7%的SubEM。
原文摘要 · Abstract (English)
Advancements in Large Language Models (LLMs) have extended their input context length, yet they still struggle with retrieval and reasoning in long-context inputs. Existing methods propose to utilize the prompt strategy and retrieval head to alleviate this limitation. However, they still face challenges in balancing retrieval precision and recall, impacting their efficacy in answering questions. To address this, we introduce $\textbf{CAFE}$, a two-stage coarse-to-fine method to enhance multi-document question-answering capacities. By gradually eliminating the negative impacts of background and distracting documents, CAFE makes the responses more reliant on the evidence documents. Initially, a coarse-grained filtering method leverages retrieval heads to identify and rank relevant documents. Then, a fine-grained steering method guides attention to the most relevant content. Experiments across benchmarks show CAFE outperforms baselines, achieving up to 22.1% and 13.7% SubEM improvement over SFT and RAG methods on the Mistral model, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。