小模型也能当评估裁判，效果接近大模型。
SLMJury: Can Small Language Models Judge as Well as Large Ones?
- 用小模型做评判，分闭合与开放两种任务模式。
- 0.6B到14B参数的小模型在数学题上快速判断更准，通用任务需长推理。
- 不同模型家族在跨领域表现差异大，无绝对赢家。
大型语言模型广泛用于评估模型输出，但其高成本、高延迟和不透明性限制了可扩展性。我们提出SLMJury框架，评估小型语言模型（SLMs）在两类判别范式下的表现：闭合式二分类正确性与开放式质量评分。我们在十个基准上测试了16个来自四个模型家族的SLM裁判（参数规模0.6B-14B），涵盖八项闭合式任务（数学、科学、通用推理，每配置64,824次判断），以及SummEval和MT-Bench用于摘要与对话评分。我们将评判建模为预算约束函数，研究五个维度。发现：(1) 过度思考效应具有领域依赖性：多数裁判在数学任务中仅用10个词快速判断即可达到或超过长推理效果（提升2-7%），而在通用任务中长推理最高可提升23%；(2) 领域泛化能力区分模型家族，数学到通用任务准确率差距从不足10%到近40%；(3) 闭合与开放式评判依赖不同能力：最佳二分类裁判Phi-4在MT-Bench上排名第九，而训练于推理的模型则反转此排序；(4) 在反射-批判-修正（RCR）辩论协议下，多代理辩论降低所有配置准确率，而顶尖裁判可抵御六种对抗人格影响，方差≤0.55%。可靠的自动化评估无需依赖大模型，但尚无单一模型全面领先。排行榜见https://anishh15.github.io/SLMJury/，框架代码及pip包已公开于https://github.com/anishh15/SLMJury和https://pypi.org/project/slmjury/。
原文摘要 · Abstract (English)
Large language models (LLMs) are widely used as judges for evaluating model outputs, but their high cost, latency, and opacity limit scalability. We introduce SLMJury, a framework for evaluating small language models (SLMs) as judges across two paradigms: closed-ended binary correctness and open-ended quality scoring. We benchmark 16 SLM judges (0.6B-14B parameters) from four model families across ten benchmarks: eight closed-ended tasks spanning mathematical, scientific, and general reasoning (N=64,824 judgments per configuration), plus SummEval and MT-Bench for summarization and conversational scoring. We formalize judging as a budget-conditioned function and study five dimensions. Four findings emerge. (1) The overthinking effect is domain-dependent: for most judges quick 10-token verdicts match or beat extended reasoning on mathematical judging (by 2-7% where they help), while reasoning wins on general tasks by up to 23%. (2) Domain generalization separates model families, with math-to-general accuracy gaps ranging from under 10% to nearly 40%. (3) Closed-ended and open-ended judging draw on different capabilities: the best binary judge (Phi-4) drops to rank 9 on MT-Bench, while reasoning-trained models invert this ordering. (4) Under the Reflect-Critique-Refine (RCR) debate protocol, multi-agent debate degrades accuracy across all tested configurations, whereas the top judges resist six adversarial personas with <=0.55% variance. Reliable automated evaluation does not require large proprietary models, yet no single SLM dominates. The leaderboard is available at https://anishh15.github.io/SLMJury/, and our framework code and pip package are publicly available at https://github.com/anishh15/SLMJury and https://pypi.org/project/slmjury/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。