arXiv:2506.15215cs.CL2025-06ACL被引 2

区分事实类与非事实类问题,实现更精准的开放问答评估

MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs

  • 先分类再评分:针对不同问题类型采用差异化评估策略
  • 在多个数据集上优于传统方法,与人工标注更一致
  • 适合需要可解释评估的LLM研究者和评测人员

开放问答是评估大语言模型能力的关键任务。相比封闭式问答,它要求更长的回答、更细致的推理过程和多样化的表达方式,因此需要精细且可解释的自动评估方法。传统指标如ROUGE和BERTScore因模型回答与参考答案模式差异大,难以捕捉语义相似性。现有基于LLM的评估方法(如成对或列表比较)缺乏直观可解释性;尽管点对点打分提供部分描述,但无法适应不同问题内容。尤其忽视了事实类与非事实类问题的区别。为此,我们提出MinosEval:先区分开放问答类型,再用不同策略排序候选答案。针对事实类问题采用自适应关键点评分,非事实类则使用实例感知的列表排名策略。在多个开放问答数据集(含自建数据集,提供更多候选答案以补充社区资源)上的实验表明,MinosEval更贴近人工标注,结果更具可解释性。

原文摘要 · Abstract (English)

Open-ended question answering (QA) is a key task for evaluating the capabilities of large language models (LLMs). Compared to closed-ended QA, it demands longer answer statements, more nuanced reasoning processes, and diverse expressions, making refined and interpretable automatic evaluation both crucial and challenging. Traditional metrics like ROUGE and BERTScore struggle to capture semantic similarities due to different patterns between model responses and reference answers. Current LLM-based evaluation approaches, such as pairwise or listwise comparisons of candidate answers, lack intuitive interpretability. While pointwise scoring of each response provides some descriptions, it fails to adapt across different question contents. Most notably, existing methods overlook the distinction between factoid and non-factoid questions. To address these challenges, we propose \textbf{MinosEval}, a novel evaluation method that first distinguishes open-ended questions and then ranks candidate answers using different evaluation strategies. For factoid questions, it applies an adaptive key-point scoring strategy, while for non-factoid questions, it uses an instance-aware listwise ranking strategy. Experiments on multiple open-ended QA datasets, including self-built ones with more candidate responses to complement community resources, show that MinosEval better aligns with human annotations and offers more interpretable results.

开放问答LLM评估可解释性分类评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。