用低成本方法优化大模型评分器，效果超现有基准。
Tuning LLM Judge Design Decisions for 1/1000 of the Cost
- 系统性调优评分器超参数，分离变量避免混淆。
- 多目标多保真搜索降低99.9%评估成本，仍保持高准确率。
- 使用开源模型，提升可复现性，适合资源有限的研究者。
评估大语言模型通常依赖昂贵的人工标注。为解决此问题，已有基于大模型的评分器方法，通过比较两个大模型输出实现无需人工干预的模型排序。然而，不同研究间存在诸多混杂因素：模型、提示词及超参数常同时变化，难以进行公平比较。本文提出系统分析与调优大模型评分器超参数的方法。为缓解评分器评估的高成本，引入多目标多保真优化，可在保证准确率的前提下显著降低搜索成本。所识别出的评分器不仅在准确率与成本效率上优于现有基准，且采用开放权重模型，提升可访问性与可复现性。实验代码已公开于 https://github.com/geoalgo/judgetuning。
原文摘要 · Abstract (English)
Evaluating Large Language Models (LLMs) often requires costly human annotations. To address this, LLM-based judges have been proposed, which compare the outputs of two LLMs enabling the ranking of models without human intervention. While several approaches have been proposed, many confounding factors are present between different papers. For instance the model, the prompt and other hyperparameters are typically changed at the same time making apple-to-apple comparisons challenging. In this paper, we propose to systematically analyze and tune the hyperparameters of LLM judges. To alleviate the high cost of evaluating a judge, we propose to leverage multi-objective multi-fidelity which allows to find judges that trade accuracy for cost and also significantly reduce the cost of the search. Our method identifies judges that not only outperform existing benchmarks in accuracy and cost-efficiency but also utilize open-weight models, ensuring greater accessibility and reproducibility. The code to reproduce our experiments is available at this repository https://github.com/geoalgo/judgetuning .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。