用大模型列表排序法，更准评估非事实类问答答案质量。
LINKAGE: Listwise Ranking among Varied-Quality References for Non-Factoid QA Evaluation via LLMs
- 将候选答案按质量排序，用大模型进行列表级评分
- 在三个数据集上与人工标注相关性显著更高
- 适合无标准答案的开放型问答评估场景
非事实类问题回答（NFQA）因答案多样且缺乏客观标准，难以评估。传统自动评价指标如ROUGE或BERTScore难以准确衡量语义相似性或不同视角的答案。近期，大语言模型（LLMs）因其在多种NLP任务中的表现被用于NFQA评估，常见方法包括对每个候选答案进行点对点打分，或在答案对间进行两两比较。受学习排序中从点对点到成对再到列表级演进的启发，我们提出一种新型列表级NFQA评估方法，利用大模型对一组参考答案按质量从高到低排序。针对无多级标准答案或无黄金答案的非事实问题，我们借助大模型生成不同质量层级的参考答案列表，以支持列表级评估。在ANTIQUES、TREC-DL-NF和WebGLM三个NFQA数据集上的实验表明，该方法与人工标注的相关性显著高于现有自动评分及常见的点对点和成对方法。
原文摘要 · Abstract (English)
Non-Factoid (NF) Question Answering (QA) is challenging to evaluate due to diverse potential answers and no objective criterion. The commonly used automatic evaluation metrics like ROUGE or BERTScore cannot accurately measure semantic similarities or answers from different perspectives. Recently, Large Language Models (LLMs) have been resorted to for NFQA evaluation due to their compelling performance on various NLP tasks. Common approaches include pointwise scoring of each candidate answer and pairwise comparisons between answers. Inspired by the evolution from pointwise to pairwise to listwise in learning-to-rank methods, we propose a novel listwise NFQA evaluation approach, that utilizes LLMs to rank candidate answers in a list of reference answers sorted by descending quality. Moreover, for NF questions that do not have multi-grade or any golden answers, we leverage LLMs to generate the reference answer list of various quality to facilitate the listwise evaluation. Extensive experimental results on three NFQA datasets, i.e., ANTIQUE, the TREC-DL-NF, and WebGLM show that our method has significantly higher correlations with human annotations compared to automatic scores and common pointwise and pairwise approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。