用注意力机制让大模型自动找长文档重点，又快又准。
AttentionRetriever: Attention Layers are Secretly Long Document Retrievers
- 利用注意力机制和实体检索构建上下文感知的文档表征
- 在长文档检索任务上显著超越现有模型，效率与密集检索相当
- 适合需要精准定位长文本信息的问答与摘要场景
检索增强生成（RAG）被广泛用于帮助大语言模型处理涉及长文档的任务。然而，现有检索模型并未针对长文档检索设计，难以应对上下文感知、因果依赖和检索范围等关键挑战。本文提出 AttentionRetriever，一种新型长文档检索模型，通过注意力机制与基于实体的检索，构建长文档的上下文感知嵌入，并确定检索范围。大量实验表明，AttentionRetriever 在长文档检索数据集上显著优于现有模型，同时保持与密集检索模型相当的效率。
原文摘要 · Abstract (English)
Retrieval augmented generation (RAG) has been widely adopted to help Large Language Models (LLMs) to process tasks involving long documents. However, existing retrieval models are not designed for long document retrieval and fail to address several key challenges of long document retrieval, including context-awareness, causal dependence, and scope of retrieval. In this paper, we proposed AttentionRetriever, a novel long document retrieval model that leverages attention mechanism and entity-based retrieval to build context-aware embeddings for long document and determine the scope of retrieval. With extensive experiments, we found AttentionRetriever is able to outperform existing retrieval models on long document retrieval datasets by a large margin while remaining as efficient as dense retrieval models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。