GaRAGe benchmark精准评估大模型在检索增强生成中识别相关依据的能力。
GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluation
- 构建含3.5万条标注段落的长答案数据集，覆盖真实场景下的检索与生成任务。
- 主流大模型在事实准确性上最高仅60%，对无关信息过度总结问题严重。
- 特别考验模型对时效性问题和私有数据源的处理能力，适合评估真实RAG系统性能。
我们提出GaRAGe，一个大规模检索增强生成（RAG）基准，包含人工标注的长篇回答及每段依据的标注，支持对大模型是否能准确识别相关依据进行细粒度评估。该基准涵盖2366个不同复杂度、动态性和主题的问题，涉及超过3.5万条从私有文档和网络中检索到的标注段落，真实反映实际RAG应用场景。它可用于评估大模型能否仅基于必要相关信息生成回答，或在信息不足时主动拒绝回答。在GaRAGe上对多个先进大模型的评测显示，模型倾向于过度总结，而非严格依据标注的相关段落（事实性得分最高仅60%），或在无相关信息时正确拒答（拒答真阳性率最高仅31%）。来源归因的F1值最高为58.9%，且在处理时间敏感问题以及从稀疏私有知识源中提取信息时表现显著下降。
原文摘要 · Abstract (English)
We present GaRAGe, a large RAG benchmark with human-curated long-form answers and annotations of each grounding passage, allowing a fine-grained evaluation of whether LLMs can identify relevant grounding when generating RAG answers. Our benchmark contains 2366 questions of diverse complexity, dynamism, and topics, and includes over 35K annotated passages retrieved from both private document sets and the Web, to reflect real-world RAG use cases. This makes it an ideal test bed to evaluate an LLM's ability to identify only the relevant information necessary to compose a response, or provide a deflective response when there is insufficient information. Evaluations of multiple state-of-the-art LLMs on GaRAGe show that the models tend to over-summarise rather than (a) ground their answers strictly on the annotated relevant passages (reaching at most a Relevance-Aware Factuality Score of 60%), or (b) deflect when no relevant grounding is available (reaching at most 31% true positive rate in deflections). The F1 in attribution to relevant sources is at most 58.9%, and we show that performance is particularly reduced when answering time-sensitive questions and when having to draw knowledge from sparser private grounding sources.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。