用大模型+评分标准，自动评估文本质量并精准预测人类评分。
LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts
- 构建多维度评分标准，让大模型逐项回答并生成评分分布。
- 通过小网络融合多个模型输出，预测人类评分，整体误差低于0.5。
- 适合评估对话系统、论文等需多维打分的场景。
本文提出一种自动化自然语言文本评估框架。基于人工构建的评分标准，大语言模型（LLM）针对每个评分维度生成可能答案的分布。尽管单个模型预测与人类评分存在偏差，且人类之间也不完全一致，但通过将多个模型的分布联合建模，可有效预测每位人类评委对所有问题的标注结果，包括综合质量或相关性评分。该方法采用一个包含评委特异性和通用参数的小型前馈神经网络进行训练。在人机信息检索对话任务中，使用9个维度（如自然性、简洁性、引用质量）的评估，该方法对1-4分制的整体用户满意度预测，均方根误差小于0.5，相比未校准基线提升2倍。
原文摘要 · Abstract (English)
This paper introduces a framework for the automated evaluation of natural language texts. A manually constructed rubric describes how to assess multiple dimensions of interest. To evaluate a text, a large language model (LLM) is prompted with each rubric question and produces a distribution over potential responses. The LLM predictions often fail to agree well with human judges -- indeed, the humans do not fully agree with one another. However, the multiple LLM distributions can be $\textit{combined}$ to $\textit{predict}$ each human judge's annotations on all questions, including a summary question that assesses overall quality or relevance. LLM-Rubric accomplishes this by training a small feed-forward neural network that includes both judge-specific and judge-independent parameters. When evaluating dialogue systems in a human-AI information-seeking task, we find that LLM-Rubric with 9 questions (assessing dimensions such as naturalness, conciseness, and citation quality) predicts human judges' assessment of overall user satisfaction, on a scale of 1--4, with RMS error $< 0.5$, a $2\times$ improvement over the uncalibrated baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。