对比RAG与GraphRAG在文本任务中的表现,提出统一评估框架
RAG vs. GraphRAG: A Systematic Evaluation and Key Insights
- 构建统一评估协议,标准化数据与配置以公平比较
- 发现RAG在问答中更稳定,GraphRAG在推理任务中优势明显
- 提供融合两者的策略,提升整体性能并揭示评估盲点
检索增强生成(RAG)通过从外部源检索信息来提升大语言模型表现,广泛应用于文本任务。针对结构化数据(如知识图谱),图检索增强生成(GraphRAG)沿图结构检索并聚合信息。近期,GraphRAG被拓展至通用文本场景,将非结构化文本组织为图表示,展现出推理与定位潜力。然而,现有文本型GraphRAG系统多针对特定任务、数据集和设计定制,导致评估协议高度异质。因此,对RAG与GraphRAG在主流文本基准上的相对优劣、局限与权衡缺乏系统理解。本文在经典文本任务(问答与查询摘要)上开展全面基准研究,提出统一评估协议,涵盖数据预处理、检索配置与生成设置,实现公平可复现的对比。结果表明,两者在不同任务与评估视角下各有优势。基于此,我们探索了两者的选型与融合策略,带来持续性能提升。进一步分析失败模式、效率权衡与评估偏差,提出设计与评估检索增强生成系统的关键考量。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) improves large language models (LLMs) by retrieving relevant information from external sources and has been widely adopted for text-based tasks. For structured data, such as knowledge graphs, Graph Retrieval-Augmented Generation (GraphRAG) retrieves and aggregates information along graph structures. More recently, GraphRAG has been extended to general text settings by organizing unstructured text into graph representations, showing promise for reasoning and grounding. Despite these advances, existing GraphRAG systems for text data are often tailored to specific tasks, datasets, and system designs, resulting in heterogeneous evaluation protocols. Consequently, a systematic understanding of the relative strengths, limitations, and trade-offs between RAG and GraphRAG on widely used text benchmarks remains limited. In this paper, we present a comprehensive benchmark study comparing RAG and GraphRAG on established text-based tasks, including question answering and query-based summarization. We introduce a unified evaluation protocol that standardizes data preprocessing, retrieval configurations, and generation settings, enabling fair and reproducible comparisons. Our results highlight the distinct strengths of RAG and GraphRAG across different tasks and evaluation perspectives. Building on these findings, we explore selection and integration strategies that combine the strengths of both paradigms, leading to consistent performance improvements. We further analyze failure modes, efficiency trade-offs, and evaluation biases, and highlight key considerations for designing and evaluating retrieval-augmented generation systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。