用层次分析法提升大模型对开放问答的评分准确性
AHP-Powered LLM Reasoning for Multi-Criteria Evaluation of Open-Ended Responses
- 结合大模型生成评价标准，再用层次分析法量化评分
- 在4个数据集上比基线更接近人类判断
- 适合需要多维度评估回答质量的研究场景
开放式问题的回答多样且难以量化，无法像封闭式问题一样简单判断对错。尽管大语言模型在各类任务中表现强劲，但在评估开放式答案时仍显不足。本文提出一种融合大语言模型与层次分析法（AHP）的方法：先由大模型生成多个评价标准，再在每个标准下通过两两比较计算各答案得分。实验在四个数据集上使用ChatGPT-3.5-turbo和GPT-4进行，结果表明该方法比四种基线更贴近人工评判。同时，研究了评价标准数量、模型差异及数据集变化对结果的影响。
原文摘要 · Abstract (English)
Question answering (QA) tasks have been extensively studied in the field of natural language processing (NLP). Answers to open-ended questions are highly diverse and difficult to quantify, and cannot be simply evaluated as correct or incorrect, unlike close-ended questions with definitive answers. While large language models (LLMs) have demonstrated strong capabilities across various tasks, they exhibit relatively weaker performance in evaluating answers to open-ended questions. In this study, we propose a method that leverages LLMs and the analytic hierarchy process (AHP) to assess answers to open-ended questions. We utilized LLMs to generate multiple evaluation criteria for a question. Subsequently, answers were subjected to pairwise comparisons under each criterion with LLMs, and scores for each answer were calculated in the AHP. We conducted experiments on four datasets using both ChatGPT-3.5-turbo and GPT-4. Our results indicate that our approach more closely aligns with human judgment compared to the four baselines. Additionally, we explored the impact of the number of criteria, variations in models, and differences in datasets on the results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。