arXiv:2511.15958cs.AIcs.CL2025-11被引 2

用多智能体协作提升小模型的推理判断能力,让小模型也能当好裁判。

JudgeBoard: Benchmarking and Enhancing Small Language Models for Reasoning Evaluation

  • 让小模型直接判断答案对错,不依赖对比或标签
  • 多智能体框架使小模型判断准确率接近甚至超过大模型
  • 适合想低成本评估推理能力的研究者和开发者

尽管小型语言模型(SLMs)在各类推理任务中表现良好,但其判断答案正确性的能力仍不如大型语言模型(LLMs)。以往基于LLM作为裁判的方法通常依赖预定义指标(如蕴含关系)比较候选答案与标准答案,这种方式间接且难自动化,难以实现细粒度、可扩展的推理输出评估。本文提出JudgeBoard,一种新型评估流程,直接向模型提问以判断答案正确性,无需额外比对。聚焦数学推理和科学/常识推理两大领域,在五个基准数据集上构建任务特定评估排行榜,采用基于准确率的排名与基于Elo的评分系统,实现模型作为裁判的一致性比较。为提升轻量级模型的判断性能,提出MAJ(多智能体判别)框架,利用具有不同推理特征的多个相互协作的SLMs,通过集体讨论逼近LLM级别的判断准确率。实验表明,孤立的SLMs在判断任务中与LLMs存在显著差距;但经由MAJ框架优化后,其可靠性与一致性大幅提高。在MATH数据集上,使用较小模型作为主干的MAJ系统表现相当甚至优于更大模型。结果表明,多智能体小模型系统有望在判断任务中达到或超越大模型表现,对可扩展、高效评估具有重要意义。

原文摘要 · Abstract (English)

While small language models (SLMs) have shown promise on various reasoning tasks, their ability to judge the correctness of answers remains unclear compared to large language models (LLMs). Prior work on LLM-as-a-judge frameworks typically relies on comparing candidate answers against ground-truth labels or other candidate answers using predefined metrics like entailment. However, this approach is inherently indirect and difficult to fully automate, offering limited support for fine-grained and scalable evaluation of reasoning outputs. In this work, we propose JudgeBoard, a novel evaluation pipeline that directly queries models to assess the correctness of candidate answers without requiring extra answer comparisons. We focus on two core reasoning domains: mathematical reasoning and science/commonsense reasoning, and construct task-specific evaluation leaderboards using both accuracy-based ranking and an Elo-based rating system across five benchmark datasets, enabling consistent model comparison as judges rather than comparators. To improve judgment performance in lightweight models, we propose MAJ (Multi-Agent Judging), a novel multi-agent evaluation framework that leverages multiple interacting SLMs with distinct reasoning profiles to approximate LLM-level judgment accuracy through collaborative deliberation. Experimental results reveal a significant performance gap between SLMs and LLMs in isolated judging tasks. However, our MAJ framework substantially improves the reliability and consistency of SLMs. On the MATH dataset, MAJ using smaller-sized models as backbones performs comparatively well or even better than their larger-sized counterparts. Our findings highlight that multi-agent SLM systems can potentially match or exceed LLM performance in judgment tasks, with implications for scalable and efficient assessment.

推理评估小模型多智能体评判框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。