用少量人工评价生成评分标准，让大模型更准更可信地评估文本生成质量。
HypoEval: Hypothesis-Guided Evaluation for Natural Language Generation
- 基于少量人工评价生成详细评分维度，再由大模型逐项打分
- 仅需30次人工标注，相关性比G-Eval高11.86%，比微调模型高11.95%
- 提供可解释的评分过程，适合需要透明评估的NLP研究与应用
大型语言模型在自动化自然语言生成评估方面展现出巨大潜力。现有基于大模型作为评判者的框架存在两方面不足：要么采用零样本设置，未参考人类意见，导致对齐度低；要么需在标注数据上微调模型，消耗大量样本。此外，多数方法缺乏评估背后的推理依据。本文提出HypoEval——一种假设引导的评估框架，先利用少量人工评估生成更详细的评分标准，再通过类似检查表的方式，将大模型在各分解维度上的打分综合为整体得分。仅需30次人工评价，HypoEval在人类排名（斯皮尔曼相关）和人类评分（皮尔逊相关）上均达到当前最佳表现，平均优于G-Eval 11.86%，且超过至少三倍人工标注量训练的微调Llama-3.1-8B-Instruct模型11.95%。我们还系统性评估了HypoEval的鲁棒性，验证其作为可靠、可解释自动化评估框架的有效性。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated great potential for automating the evaluation of natural language generation. Previous frameworks of LLM-as-a-judge fall short in two ways: they either use zero-shot setting without consulting any human input, which leads to low alignment, or fine-tune LLMs on labeled data, which requires a non-trivial number of samples. Moreover, previous methods often provide little reasoning behind automated evaluations. In this paper, we propose HypoEval, Hypothesis-guided Evaluation framework, which first uses a small corpus of human evaluations to generate more detailed rubrics for human judgments and then incorporates a checklist-like approach to combine LLM's assigned scores on each decomposed dimension to acquire overall scores. With only 30 human evaluations, HypoEval achieves state-of-the-art performance in alignment with both human rankings (Spearman correlation) and human scores (Pearson correlation), on average outperforming G-Eval by 11.86% and fine-tuned Llama-3.1-8B-Instruct with at least 3 times more human evaluations by 11.95%. Furthermore, we conduct systematic studies to assess the robustness of HypoEval, highlighting its effectiveness as a reliable and interpretable automated evaluation framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。