提出新基准GroUSE，检验LLM评估模型对文本生成错误的识别能力。
GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answering
- 构建144个单元测试，覆盖7类常见生成错误。
- 发现现有评估框架常遗漏关键错误，即使使用GPT-4也如此。
- 建议用单元测试补全对齐GPT-4的评估，提升错误检测精度。
检索增强生成（RAG）已成为结合大语言模型（LLMs）与私有、实时知识库的通用范式。本文针对使用LLM作为评判者评估RAG生成答案时的挑战，识别出7类生成器失效模式，提出GroUSE（基于问答的评判者统一评分基准），包含144个单元测试。该基准揭示，现有自动化RAG评估框架即便使用GPT-4作为评判者，仍常忽略重要失效模式。为改进评估设计，本文提出新流程，发现闭源模型在GroUSE上表现良好，而最先进的开源评判模型虽与GPT-4评价高度相关，却无法泛化到新标准。结果表明，与GPT-4的相关性不足以代表实际性能，需辅以单元测试进行精确错误识别。进一步实验显示，对Llama-3进行基于GPT-4推理轨迹的微调，显著提升其评估能力，在与GPT-4一致性和参考情境校准性上均有改善。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has emerged as a common paradigm to use Large Language Models (LLMs) alongside private and up-to-date knowledge bases. In this work, we address the challenges of using LLM-as-a-Judge when evaluating grounded answers generated by RAG systems. To assess the calibration and discrimination capabilities of judge models, we identify 7 generator failure modes and introduce GroUSE (Grounded QA Unitary Scoring of Evaluators), a meta-evaluation benchmark of 144 unit tests. This benchmark reveals that existing automated RAG evaluation frameworks often overlook important failure modes, even when using GPT-4 as a judge. To improve on the current design of automated RAG evaluation frameworks, we propose a novel pipeline and find that while closed models perform well on GroUSE, state-of-the-art open-source judges do not generalize to our proposed criteria, despite strong correlation with GPT-4's judgement. Our findings suggest that correlation with GPT-4 is an incomplete proxy for the practical performance of judge models and should be supplemented with evaluations on unit tests for precise failure mode detection. We further show that finetuning Llama-3 on GPT-4's reasoning traces significantly boosts its evaluation capabilities, improving upon both correlation with GPT-4's evaluations and calibration on reference situations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。