构建首个评估文本与表格联合生成的基准，挑战模型先找对数据再推理的能力。
T$^2$-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation
- 设计2.3万组问答对，要求模型先检索正确文本表格再推理
- 91.3%问题独立于上下文，确保评估结果可靠
- 发现混合检索法（Hybrid BM25）最优，但顶尖模型仍难应对
由于许多真实文档同时包含文本和表格数据，构建鲁棒的检索增强生成（RAG）系统对于有效访问和分析此类内容以支持复杂推理任务至关重要。本文提出 $T^2$-RAGBench，一个包含 23,088 个问题-上下文-答案三元组的基准，用于评估 RAG 方法在真实世界文本与表格数据上的表现。与通常在“理想上下文”设置下运行的 QA 数据集不同，$T^2$-RAGBench 要求模型首先检索正确的上下文，再进行数值推理。现有包含文本与表格数据的 QA 数据集通常包含依赖上下文的问题，可能导致多个正确答案。为此，我们将 SOTA 数据集转换为上下文无关格式，经专家验证，其中 91.3% 的问题是上下文无关的，从而实现可靠的 RAG 评估。全面评估表明，结合稠密与稀疏向量的 Hybrid BM25 是处理文本与表格数据最有效的检索方法。然而，实验结果也显示，即使使用当前最先进的大语言模型和 RAG 方法，$T^2$-RAGBench 仍极具挑战性。进一步的消融研究分析了嵌入模型与语料库规模对检索性能的影响。$T^2$-RAGBench 为现有 RAG 方法在文本与表格数据上的评估提供了真实且严格的基准。代码与数据集已公开：https://github.com/uhh-hcds/g4kmu-paper
原文摘要 · Abstract (English)
Since many real-world documents combine textual and tabular data, robust Retrieval Augmented Generation (RAG) systems are essential for effectively accessing and analyzing such content to support complex reasoning tasks. Therefore, this paper introduces $\textbf{$T^2$-RAGBench}$, a benchmark comprising $\textbf{23,088}$ question-context-answer triples, designed to evaluate RAG methods on real-world text-and-table data. Unlike typical QA datasets that operate under $\textit{Oracle Context}$ settings, $\textbf{$T^2$-RAGBench}$ challenges models to first retrieve the correct context before conducting numerical reasoning. Existing QA datasets containing text-and-table data typically contain context-dependent questions, which may yield multiple correct answers depending on the provided context. To address this, we transform SOTA datasets into a context-independent format, validated by experts as 91.3% context-independent questions, enabling reliable RAG evaluation. Our comprehensive evaluation identifies $\textit{Hybrid BM25}$ , a technique that combines dense and sparse vectors, as the most effective approach for text-and-table data. However, results demonstrate that $\textbf{$T^2$-RAGBench}$ remains challenging even for SOTA LLMs and RAG methods. Further ablation studies examine the impact of embedding models and corpus size on retrieval performance. $\textbf{$T^2$-RAGBench}$ provides a realistic and rigorous benchmark for existing RAG methods on text-and-table data. Code and dataset are available online: https://github.com/uhh-hcds/g4kmu-paper
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。