首个评估全局检索增强生成能力的基准,解决长文档全局推理难题。
Towards Global Retrieval Augmented Generation: A Benchmark for Corpus-Level Reasoning
- 构建跨文档全局推理任务框架,支持计数、极值、排序与前K提取。
- 现有方法在全局任务上表现差,最强基线仅1.51 F1得分。
- 提出GlobalRAG多工具协同框架,提升信息聚合与符号计算能力。
检索增强生成(RAG)已成为减少大语言模型幻觉的主流方法。当前评估基准主要关注局部RAG:从少量文档片段中检索相关段落以回答仅需局部理解的问题。然而,许多真实应用场景需要更复杂的全局RAG能力——即整合分析整个文档集合以获得全局洞察(例如:2023年被引用最多的前十篇论文是什么?)。本文提出GlobalQA,首个专为评估全局RAG能力设计的基准,涵盖四类核心任务:计数、极值查询、排序与前K项提取。系统评估不同模型与基线后发现,现有RAG方法在全局任务上表现不佳,最强基线仅达1.51 F1得分。为此,我们提出GlobalRAG,一种多工具协同框架:通过段落级检索保持结构连贯性,引入基于LLM的智能过滤器剔除噪声文档,并集成聚合模块实现精确符号计算。在Qwen2.5-14B模型上,GlobalRAG达到6.63 F1,显著优于基线的1.51 F1,验证了方法有效性。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) has emerged as a leading approach to reducing hallucinations in large language models (LLMs). Current RAG evaluation benchmarks primarily focus on what we call local RAG: retrieving relevant chunks from a small subset of documents to answer queries that require only localized understanding within specific text chunks. However, many real-world applications require a fundamentally different capability -- global RAG -- which involves aggregating and analyzing information across entire document collections to derive corpus-level insights (for example, "What are the top 10 most cited papers in 2023?"). In this paper, we introduce GlobalQA -- the first benchmark specifically designed to evaluate global RAG capabilities, covering four core task types: counting, extremum queries, sorting, and top-k extraction. Through systematic evaluation across different models and baselines, we find that existing RAG methods perform poorly on global tasks, with the strongest baseline achieving only 1.51 F1 score. To address these challenges, we propose GlobalRAG, a multi-tool collaborative framework that preserves structural coherence through chunk-level retrieval, incorporates LLM-driven intelligent filters to eliminate noisy documents, and integrates aggregation modules for precise symbolic computation. On the Qwen2.5-14B model, GlobalRAG achieves 6.63 F1 compared to the strongest baseline's 1.51 F1, validating the effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。