利用中间层表示提升多跳文档检索效率，性能接近多步方法但更快
Optimizing Multi-Hop Document Retrieval Through Intermediate Representations
- 提取大模型中间层的语义表示，代替反复生成新查询
- 在三个数据集上超越现有RAG方法，推理开销接近标准RAG
- 适合需要高效多跳问答的应用场景
检索增强生成（RAG）在处理复杂多跳问题时面临挑战。现有方法通过迭代生成内部查询和检索外部文档来应对，但计算开销大。本文发现大模型在逐层推理中存在提取-处理-再提取的三阶段信息处理模式，中间层表示包含更丰富的信息。基于此，提出分层RAG（L-RAG），不依赖生成新查询，而是利用中间层表示捕捉下一步所需知识进行检索。实验表明，L-RAG在开放域多跳问答数据集MuSiQue、HotpotQA、2WikiMultiHopQA上表现优于现有RAG方法，同时推理开销与标准RAG相当。代码已开源。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) encounters challenges when addressing complex queries, particularly multi-hop questions. While several methods tackle multi-hop queries by iteratively generating internal queries and retrieving external documents, these approaches are computationally expensive. In this paper, we identify a three-stage information processing pattern in LLMs during layer-by-layer reasoning, consisting of extraction, processing, and subsequent extraction steps. This observation suggests that the representations in intermediate layers contain richer information compared to those in other layers. Building on this insight, we propose Layer-wise RAG (L-RAG). Unlike prior methods that focus on generating new internal queries, L-RAG leverages intermediate representations from the middle layers, which capture next-hop information, to retrieve external knowledge. L-RAG achieves performance comparable to multi-step approaches while maintaining inference overhead similar to that of standard RAG. Experimental results show that L-RAG outperforms existing RAG methods on open-domain multi-hop question-answering datasets, including MuSiQue, HotpotQA, and 2WikiMultiHopQA. The code is available in https://github.com/Olive-2019/L-RAG
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。