低成本且抗偏见的LLM评估系统，让模型对比更公平、更省力。
Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation
- 基于最大后验估计构建灵活评分体系，可捕捉人类偏好偏差
- 新模型评估成本降41%，新任务降77%，复用已有基准数据
- 支持跨任务直接比较，全面揭示模型优劣势
基于评分的人工评估已成为准确衡量大语言模型性能的重要工具。然而，现有评分系统存在三大局限：无法考虑显著影响评估结果的偏差；需要大量昂贵的偏好数据集才能获得准确评分；难以在不同任务间进行有意义的模型评分比较。为此，我们提出Polyrating，一种基于最大后验估计的表达性强、灵活的评分系统，可在更低成本下实现对模型性能的更细致分析。Polyrating能检测并量化影响人类偏好的偏差，确保更公平的模型比较。此外，通过利用现有基准分数，可使新模型的人工评估成本降低高达41%，新任务降低高达77%。最后，Polyrating支持跨任务的直接评分比较，为理解大语言模型在不同应用中的优势、劣势及相对表现提供全面视角。
原文摘要 · Abstract (English)
Rating-based human evaluation has become an essential tool to accurately evaluate the impressive performance of large language models (LLMs). However, current rating systems suffer from several important limitations: first, they fail to account for biases that significantly influence evaluation results, second, they require large and expensive preference datasets to obtain accurate ratings, and third, they do not facilitate meaningful comparisons of model ratings across different tasks. To address these issues, we introduce Polyrating, an expressive and flexible rating system based on maximum a posteriori estimation that enables a more nuanced and thorough analysis of model performance at lower costs. Polyrating can detect and quantify biases affecting human preferences, ensuring fairer model comparisons. Further, Polyrating can reduce the cost of human evaluations by up to $41\%$ for new models and up to $77\%$ for new tasks by leveraging existing benchmark scores. Lastly, Polyrating enables direct comparisons of ratings across different tasks, providing a comprehensive understanding of an LLMs' strengths, weaknesses, and relative performance across different applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。