对比人类与大模型评估RAG生成答案的引用支持度，发现大模型表现接近人类。
Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges
- 用GPT-4o自动判断答案引用是否支持，与人工评估对比
- 无修改时匹配率达56%，后编辑后提升至72%
- 大模型判断更接近独立人类，适合做支持评估替代方案
检索增强生成（RAG）通过引用包含“真实信息”的源文档来生成答案，减少模型幻觉。评估的关键是“支持度”——引用内容是否支持答案。我们对TREC 2024 RAG Track中45个参赛系统在36个主题上的结果进行了大规模比较，评估了自动大模型裁判（GPT-4o）与人类裁判在支持度判断上的表现。考察两种情形：（1）从零开始的人工评估；（2）基于大模型预测的后编辑人工评估。结果显示，在56%的完全人工评估中，人类与GPT-4o判断完全一致（三分类尺度），在后编辑条件下上升至72%。进一步分析分歧发现，独立人类裁判与GPT-4o的相关性高于人类裁判自身，表明大模型可作为支持度评估的可靠替代。最后，我们对人类与GPT-4o的错误进行定性分析，为未来评估迭代提供指导。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) enables large language models (LLMs) to generate answers with citations from source documents containing "ground truth", thereby reducing system hallucinations. A crucial factor in RAG evaluation is "support", whether the information in the cited documents supports the answer. To this end, we conducted a large-scale comparative study of 45 participant submissions on 36 topics to the TREC 2024 RAG Track, comparing an automatic LLM judge (GPT-4o) against human judges for support assessment. We considered two conditions: (1) fully manual assessments from scratch and (2) manual assessments with post-editing of LLM predictions. Our results indicate that for 56% of the manual from-scratch assessments, human and GPT-4o predictions match perfectly (on a three-level scale), increasing to 72% in the manual with post-editing condition. Furthermore, by carefully analyzing the disagreements in an unbiased study, we found that an independent human judge correlates better with GPT-4o than a human judge, suggesting that LLM judges can be a reliable alternative for support assessment. To conclude, we provide a qualitative analysis of human and GPT-4o errors to help guide future iterations of support assessment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。