arXiv:2602.10210cs.LG2026-02KDD

构建新基准,评估模型在混合知识下的多跳推理能力。

How Much Reasoning Do Retrieval-Augmented Models Add beyond LLMs? A Benchmarking Framework for Multi-Hop Inference over Hybrid Knowledge

  • 自动融合arXiv论文的文本与知识图谱生成问题
  • 在三大领域验证了模型真正依赖检索而非记忆
  • 支持按时间与领域定制,适合研究混合知识系统

大语言模型(LLMs)在需要最新信息和多跳推理的知识密集型问题上仍表现不佳。将LLMs与混合外部知识(如非结构化文本和结构化知识图谱)结合,是避免昂贵持续预训练的可行方案。然而,现有基准往往与模型预训练数据重叠,导致答案或支撑知识可能已编码在模型参数中,难以区分真实检索与参数化回忆。为此,我们提出HybridRAG-Bench框架,可自动生成基于arXiv近期科学文献的非结构化文本与知识图谱耦合数据,构建具有明确推理路径的知识密集型问答对。该框架支持灵活选择领域与时间范围,实现抗污染、可定制的评估。在人工智能、治理政策与生物信息学三个领域的实验表明,HybridRAG-Bench能有效奖励真正的检索与推理,而非参数记忆,为混合知识增强推理系统提供可信测试平台。代码与数据已开源至github.com/junhongmit/HybridRAG-Bench。

原文摘要 · Abstract (English)

Large language models (LLMs) continue to struggle with knowledge-intensive questions that require up-to-date information and multi-hop reasoning. Augmenting LLMs with hybrid external knowledge, such as unstructured text and structured knowledge graphs, offers a promising alternative to costly continual pretraining. As such, reliable evaluation of their retrieval and reasoning capabilities becomes critical. However, many existing benchmarks increasingly overlap with LLM pretraining data, which means answers or supporting knowledge may already be encoded in model parameters, making it difficult to distinguish genuine retrieval and reasoning from parametric recall. We introduce HybridRAG-Bench, a framework for constructing benchmarks to evaluate retrieval-intensive, multi-hop reasoning over hybrid knowledge. HybridRAG-Bench automatically couples unstructured text and structured knowledge graph representations derived from recent scientific literature on arXiv, and generates knowledge-intensive question-answer pairs grounded in explicit reasoning paths. The framework supports flexible domain and time-frame selection, enabling contamination-aware and customizable evaluation as models and knowledge evolve. Experiments across three domains (artificial intelligence, governance and policy, and bioinformatics) demonstrate that HybridRAG-Bench rewards genuine retrieval and reasoning rather than parametric recall, offering a principled testbed for evaluating hybrid knowledge-augmented reasoning systems. We release our code and data at github.com/junhongmit/HybridRAG-Bench.

知识增强多跳推理基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。