用大模型互评+积分制,自动评估生成文本质量。
(Towards) Scalable Reliable Automated Evaluation with Large Language Models

- 多模型两两比较,用埃洛积分生成稳定排名。
- 自动评分与专家判断高度一致,减少人工干预。
- 适合作为跨领域、可扩展的自动化评估工具。
评估大语言模型生成文本的质量与相关性仍具挑战且耗时。现有自动化指标难以捕捉生成内容的复杂性与多样性,且通常依赖明确参考标准,限制了在缺乏客观基准领域的应用。本文提出一种新评估框架,通过多个大模型对输出进行成对比较,降低单个模型偏差;采用埃洛积分系统生成稳定可解释的排名。可通过调节一致阈值(从完全一致到多数投票)灵活控制评估置信度与覆盖范围。在科学摘要提取的胜任力画像评估中,初步结果显示自动排名与专家判断高度相关,显著减少了人工干预需求。该框架提供了一种可扩展、一致且领域无关的评估层,支持在多种应用场景下高效可靠地评估大模型输出质量。
原文摘要 · Abstract (English)
Evaluating the quality and relevance of textual outputs from Large Language Models (LLMs) remains challenging and resource-intensive. Existing automated metrics often fail to capture the complexity and variability inherent in LLM-generated outputs. Moreover, these metrics typically rely on explicit reference standards, limiting their use mostly to domains with objective benchmarks. This work introduces a novel evaluation framework designed to approximate expert-level assessments of LLM-generated content. The proposed method employs pairwise comparisons of outputs by multiple LLMs, reducing biases from individual models. An Elo rating system is used to generate stable and interpretable rankings. Adjustable agreement thresholds, from full unanimity to majority voting, allow flexible control over evaluation confidence and coverage. The method's effectiveness is demonstrated through evaluating competency profiles extracted from scientific abstracts. Preliminary results show that automatically derived rankings correlate well with expert judgments, significantly reducing the need for extensive human intervention. By offering a scalable, consistent, and domain-agnostic evaluation layer, the framework supports more efficient and reliable quality assessments of LLM outputs across diverse applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。