用大模型当裁判评估抽取式问答，效果比传统指标更接近真人评分。
Reassessing Extractive QA Datasets at Scale: LLM-as-a-Judge and In-Depth Analyses
- 让大模型担任评分员，跨多个数据集对比不同提示方式。
- 大模型评分与人工评分相关性高达0.85，远超传统指标。
- 对数字类答案表现好，提示词影响小，无需上下文也能精准打分。
抽取式问答常用精确匹配（EM）和F1分数评估，但这些指标常无法真实反映模型性能。近期研究提出使用大语言模型（LLM）作为评判者（LLM-as-a-judge），但缺乏在多数据集上的全面评估，且忽视答案类型敏感性、提示差异及自偏好偏差等关键因素。本文系统研究了四种抽取式问答数据集上，多种提示变体下多类大模型在问答与评判角色中的表现。结果表明，LLM-as-a-judge的评分与人类评价的相关性显著高于EM（0.22）和F1（0.40），最高可达0.85，尤其在数字类答案上表现优异；与其它NLP任务不同，未发现自偏好偏差，即使同一模型同时担任问答与评判角色也无偏倚；此外，提示语句变化影响极小，零样本、无上下文的评判反而表现最佳。
原文摘要 · Abstract (English)
Extractive QA tasks are commonly evaluated using Exact Match (EM) and F1-score, but these metrics often fail to reflect true model performance. Recent studies have proposed using large language models (LLMs) as judges (LLM-as-a-judge), yet they often lack comprehensive evaluation across datasets and overlook key factors such as sensitivity to answer types, prompt variations, and self-preference bias. In this work, we conduct a systematic study of LLM-as-a-judge across four extractive QA datasets and various prompt variations, assessing multiple LLM families in both answering and judging roles. Our results show that LLM-as-a-judge judgments correlate much more strongly with human evaluations than EM (0.22) and F1 (0.40), achieving correlations up to 0.85 with open-source models. Further analysis reveals that LLM-as-a-judge performs particularly well on number-related answers but faces challenges with more complex types, such as job titles. Contrary to findings in other NLP tasks, we observe no self-preference bias, even when the same model serves as both QA model and judge. Finally, we find that prompt phrasing has minimal impact, and zero-shot, context-free judging often yields the best evaluation performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。