用少量人工标注评估大模型评分器可靠性，节省3成成本
Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability
- 从有限标注中挑选代表性样本，匹配整体可靠性指标
- 在15个数据集上误差降18.7%，胜率0.838，标注量减32.5%
- 适合需低成本验证大模型评分器的开发者与研究者
大模型评分器可减少开放文本生成评价中的人工成本，但其可靠性依赖于与人类评分的一致性——而这种一致性本身需要昂贵的人工标注。本文提出一种名为Metric Match的方法，仅通过少量标注即可估算大模型评分器的基于相关性的可靠性指标。该方法选择一组样本进行人工标注，使该子集在合成标签下与总体可靠性指标匹配。实验表明，在四个相关性度量和15个数据集上，Metric Match相比随机选样胜率达到0.838，平均估计误差降低18.7%，标注需求减少32.5%。我们还提供成本模型，并以医疗案例说明，该方法比随机选择为专家标注节省1041.67美元。进一步地，我们将任务转为判断评分器是否超过部署阈值，结果仍优于随机选择。所有代码公开，且提供可安装包。
原文摘要 · Abstract (English)
LLM judges are used to reduce the need for costly human labor in evaluating open-ended text generation. However, the reliability of these judges depends critically on their alignment with human raters -- a property that itself depends on costly human annotations. In this work, we develop a method (Metric Match) for estimating correlation-based reliability metrics of LLM judges from limited annotations. Metric Match selects a subset of samples for human annotation such that the subset matches the population reliability metric with respect to acquired synthetic labels. We empirically show that Metric Match achieves a win-rate of 0.838 against random subset selection across four different correlation metrics and 15 datasets, with an 18.7% decrease in average estimation error and reduces annotation needs by 32.5%. We provide a cost model and highlight a medical case study where our method saves $1,041.67 compared to random selection for expert annotation. Further, we shift our task from reliability estimation to reliability classification of whether a given judge is above a deployment threshold, outperforming random selection with Metric Match. All project code is publicly available, and we additionally provide an installable package for ease of use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。