用多路聚类分析大模型评分偏见，发现评价模式结构。
MultiwayPAM: Multiway Partitioning Around Medoids for LLM-as-a-Judge Score Analysis
- 提出多路PAM聚类法，同时找评分数据的聚类中心和归属
- 在两个数据集上验证，能揭示问题/回答者/评价者间的隐藏模式
- 适合研究大模型评估偏差或优化评测框架的人看
LLM-as-a-Judge 是一种灵活的文本评估框架，通过更换提示模板可从多个角度获取文本质量评分。使用该框架面临两大挑战：大模型推理带来的高计算成本，以及大模型评价者固有的偏见。为解决这些问题并揭示大模型评价者导致的评分偏见结构，本文提出对一个包含问题、回答者和评价者组合评分的张量应用张量聚类方法。具体地，开发了一种新的张量聚类方法 MultiwayPAM，可同时估计每个模式（问题、回答者、评价者）的聚类成员关系和对应的中位数（medoids）。通过观察 MultiwayPAM 得到的中位数，可获知各聚类中问题、回答者或评价者的组成情况。实验表明，MultiwayPAM 在两个实际数据集上的评分张量中均有效，能够揭示评分偏见的内在结构。
原文摘要 · Abstract (English)
LLM-as-a-Judge is a flexible framework for text evaluation, which allows us to obtain scores for the quality of a given text from various perspectives by changing the prompt template. Two main challenges in using LLM-as-a-Judge are computational cost of LLM inference, especially when evaluating a large number of texts, and inherent bias of an LLM evaluator. To address these issues and reveal the structure of score bias caused by an LLM evaluator, we propose to apply a tensor clustering method to a given LLM-as-a-Judge score tensor, whose entries are the scores for different combinations of questions, answerers, and evaluators. Specifically, we develop a new tensor clustering method MultiwayPAM, with which we can simultaneously estimate the cluster membership and the medoids for each mode of a given data tensor. By observing the medoids obtained by MultiwayPAM, we can gain knowledge about the membership of each question/answerer/evaluator cluster. We experimentally show the effectiveness of MultiwayPAM by applying it to the score tensors for two practical datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。