评测大模型在检索增强生成中的综合能力，涵盖推理、整合、表格理解等关键技能。
LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation
- 构建五类能力评估框架，覆盖逻辑、推理、表格理解与拒答等真实场景需求。
- 测试显示所有模型整体准确率均未超90%，揭示当前LLM在复杂任务中的局限性。
- 适用于实际RAG系统选型或训练专用模型，支持中文和英文双语评估。
检索增强生成（RAG）框架中，生成器（如大语言模型）通过外部文档检索来生成答案。现实中，生成器需整合长上下文、多步推理、表格解读，并在证据缺失时拒绝回答。现有评测覆盖有限，无法统一评估多项能力。为此，我们提出LIT-RAGBench（逻辑、整合、表格、推理与拒答的RAG生成器基准），包含五大类别：整合、推理、逻辑、表格与拒答，每类细分为实际评估维度。该基准系统化涵盖跨类别组合模式。通过虚构实体与情景，确保答案基于所提供文档。数据集包含114个人工构建的日文问题及机器翻译后经人工校对的英文版本。采用LLM-as-a-Judge评分，报告各类别与整体准确率。在基于API和开源权重的模型中，无一模型整体准确率超过90%。通过量化各类别优劣，LIT-RAGBench可为实际RAG部署提供模型选择依据，并推动专用于RAG的模型开发。相关数据集与评估代码已公开于https://github.com/Koki-Itai/LIT-RAGBench。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) is a framework in which a Generator, such as a Large Language Model (LLM), produces answers by retrieving documents from an external collection using a Retriever. In practice, Generators must integrate evidence from long contexts, perform multi-step reasoning, interpret tables, and abstain when evidence is missing. However, existing benchmarks for Generators provide limited coverage, with none enabling simultaneous evaluation of multiple capabilities under unified conditions. To bridge the gap between existing evaluations and practical use, we introduce LIT-RAGBench (the Logic, Integration, Table, Reasoning, and Abstention RAG Generator Benchmark), which defines five categories: Integration, Reasoning, Logic, Table, and Abstention, each further divided into practical evaluation aspects. LIT-RAGBench systematically covers patterns combining multiple aspects across categories. By using fictional entities and scenarios, LIT-RAGBench evaluates answers grounded in the provided external documents. The dataset consists of 114 human-constructed Japanese questions and an English version generated by machine translation with human curation. We use LLM-as-a-Judge for scoring and report category-wise and overall accuracy. Across API-based and open-weight models, no model exceeds 90% overall accuracy. By making strengths and weaknesses measurable within each category, LIT-RAGBench serves as a valuable metric for model selection in practical RAG deployments and for building RAG-specialized models. We release LIT-RAGBench, including the dataset and evaluation code, at https://github.com/Koki-Itai/LIT-RAGBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。