用大模型动态控制检索,实现长文档问答的高效高准。
PECAN: LLM-Guided Dynamic Progress Control with Attention-Guided Hierarchical Weighted Graph for Long-Document QA
- 用大模型实时调节检索量,平衡效果与效率。
- 基于注意力构建分层图,提升关键信息召回率。
- 在多个数据集上达到大模型精度,计算开销接近传统检索方法。
长文档问答面临文本规模大、长距离依赖等问题。尽管大语言模型(LLMs)可单次处理全文,但计算成本过高。检索增强生成(RAG)虽通过分块提升效率,却常损失全局上下文且表现不佳。现有结合LLM与RAG的迭代摘要方法或未能充分使用LLM能力,或仍存在高计算开销。本文提出PECAN:一种基于大模型引导的动态进度控制与注意力引导的分层加权图检索方法。核心改进包括:(1) 大模型引导的动态进度控制:利用大模型根据查询动态调整检索内容量,实现效果与效率的更好平衡;(2) 基于注意力的检索:构建分层图结构,边权重由大模型注意力得分决定。实验表明,PECAN在两个单文档与两个多文档问答数据集上达到大模型级性能,同时计算复杂度与RAG方法相当。
原文摘要 · Abstract (English)
Long-document QA presents challenges with large-scale text and long-distance dependencies. Recent advances in Large Language Models (LLMs) enable entire documents to be processed in a single pass. However, their computational cost is significantly high. Retrieval-Augmented Generation (RAG) methods split text into smaller chunks, but they often yield inferior results and may lose global context. Recent approaches that integrate LLMs into RAG via iterative summarization either underutilize LLM capabilities or still incur high computational costs. In this paper, we combine the high accuracy of LLMs with the efficiency of RAG and propose LLM-Guided Dynamic Progress Control with Attention-Based Hierarchical Weighted Graph (PECAN). Our method introduces two key improvements: (1) LLM-Guided Dynamic Progress Control: We leverage LLMs to dynamically control the retrieval process, adjusting the amount of retrieved information based on different queries to achieve a better balance of effectiveness and efficiency. (2) Attention-Guided Retrieval: We propose a novel retrieval method that constructs a hierarchical graph where edges are derived by LLM attention weights. Experimental results demonstrate that PECAN achieves LLM-level performance while maintaining computational complexity comparable to that of RAG methods on two single-document and two multi-document QA datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。