统一LLM评判框架,让模型评估更透明可复现。
JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
- 整合四大主流评测基准,支持更换评判模型和记录完整日志。
- 开源模型评判配置在多语言下媲美闭源模型性能。
- 用少量人工标注+大模型评判,低成本模拟高精度评分结果。
LLM作为评判者已成为语言模型排序的主流范式,但当前生态碎片化:多数基准自带代码库、硬编码特定闭源评判模型,且仅支持单一评估协议。这种碎片化使得难以研究设计选择(如基准、评判模型、提示词、推理后端)如何影响对模型质量的判断。我们提出JudgeArena,一个开源框架,将AlpacaEval、Arena-Hard、MT-Bench和m-Arena-Hard等主要LLM评判基准统一于单一接口,支持可替换评判模型并全面记录元数据,提升报告透明度与可复现性。该框架允许任何可通过vLLM、llama.cpp或OpenRouter访问的模型充当候选或评判者。此外,JudgeArena提供了针对开放模型优化的评判配置,在英文及多语言人类偏好数据集上表现匹配甚至超越闭源模型,减少对不透明闭源模型的依赖。最后,结合现有人工标注与目标模型的LLM评判结果,JudgeArena可高精度模拟LMArena Elo分数,为大规模人工标注提供一种实用、开放且低成本的替代方案。
原文摘要 · Abstract (English)
LLM-as-a-judge evaluation has become a dominant paradigm for ranking language models, yet the ecosystem remains fragmented: most benchmarks ship their own code base, hardcode a specific closed-model judge, and support a single evaluation protocol. This fragmentation makes it difficult to study how design choices--the benchmark, the judge model, the prompt, the inference backend--affect the conclusions we draw about model quality. We introduce JudgeArena, an open-source framework that unifies major LLM-judge benchmarks (AlpacaEval, Arena-Hard, MT-Bench, and m-Arena-Hard) under a single interface with swappable judges and comprehensive metadata logging for increased transparency in reporting and reproducibility. It enables systematic studies of judge choices, as any model accessible via vLLM, llama.cpp, or OpenRouter can serve as both candidate and judge. Furthermore, JudgeArena ships with tuned judge configurations for open models that match or outperform closed-model judges, validated on human preference datasets in both English and multilingual settings, reducing the reliance on opaque closed models. Finally, by combining existing human annotations with LLM-judge evaluations of a target model, JudgeArena can simulate LMArena Elo scores with high accuracy offering a practical, open, and low-cost alternative to large-scale human annotation campaigns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。