用模块化推理提升大模型评分可靠性，效果媲美超大模型。
Verdict: A Library for Scaling Judge-Time Compute
- 通过组合验证、辩论等模块，动态增强评分时的计算能力。
- 在内容审核等任务上，性能媲美参数量大数十倍的微调模型。
- 适合需要可解释、高可靠评分系统的研究人员与开发者。
使用大语言模型作为自动评判者（LLM-as-a-judge）已广泛应用，但传统评判者存在诸多可靠性问题。为此，我们提出 Verdict——一个开源库，用于扩展评判时的计算资源，以提升自动化评估系统的准确性、可靠性与可解释性。Verdict 通过组合模块化推理单元（如验证、辩论、聚合）并增加推理时的计算开销，显著改善 LLM 判决质量。在内容审核、事实核查和幻觉检测等挑战性任务中，Verdict 判决者的性能达到与参数量大数十倍的微调模型、提示模型及推理模型相当的水平。该框架为研究者与实践者构建可扩展、可解释、可靠的 LLM 评估系统奠定了基础。
原文摘要 · Abstract (English)
The use of LLMs as automated judges ("LLM-as-a-judge") is now widespread, yet standard judges suffer from a multitude of reliability issues. To address these challenges, we introduce Verdict, an open-source library for scaling judge-time compute to enhance the accuracy, reliability, and interpretability of automated evaluators. Verdict leverages the composition of modular reasoning units (such as verification, debate, and aggregation) and increased inference-time compute to improve LLM judge quality. Across a variety of challenging tasks such as content moderation, fact-checking, and hallucination detection, Verdict judges achieves performance competitive with orders-of-magnitude larger fine-tuned judges, prompted judges, and reasoning models. Our framework establishes a foundation for scalable, interpretable, and reliable LLM-based evaluation systems for both researchers and practitioners.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。