用可执行程序替代大模型评分，实现低成本高可靠评估
Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluation
- 让大模型生成可运行的评判程序，而非直接打分
- 本地运行成本降为原来的千分之一，判断一致性提升15.83%
- 适合需要可解释、可复现评估的科研与工程场景
大型语言模型（LLMs）被广泛用于评估生成内容的质量，但存在高昂的API成本、可靠性不确定、流程僵化及固有偏见等问题。为此，我们提出PAJAMA（Program-As-a-Judge for Automated Model Assessment），通过让大模型生成可执行的评判程序来替代直接评分。这些程序可本地存储并运行，成本降低三个数量级，同时提供可解释、可审计的判断逻辑，易于调整。相比基于Qwen2.5-14B的LLM-as-a-judge，PAJAMA使判断一致性提升15.83%，偏见响应减少23.7%。将程序判别结果蒸馏为模型后，其在RewardBench的CHAT-HARD子集上表现更优：在Prometheus上领先2.19%，在JudgeLM数据集上领先8.67%，且成本仅为前者的千分之一。
原文摘要 · Abstract (English)
Large language models (LLMs) are widely used to evaluate the quality of LLM generations and responses, but this leads to significant challenges: high API costs, uncertain reliability, inflexible pipelines, and inherent biases. To address these, we introduce PAJAMA (Program-As-a-Judge for Automated Model Assessment), a new alternative that uses LLMs to synthesize executable judging programs instead of directly scoring responses. These synthesized programs can be stored and run locally, costing orders of magnitude less while providing interpretable, and auditable judging logic that can be easily adapted. Program-based judges mitigate biases, improving judgment consistency by 15.83% and reducing biased responses by 23.7% on average compared to a Qwen2.5-14B-based LLM-as-a-judge. When program judgments are distilled into a model, PAJAMA outperforms LLM-as-a-judge on the challenging CHAT-HARD subset of RewardBench, outperforming metrics by 2.19% on Prometheus and 8.67% on the JudgeLM dataset, all at three orders of magnitude lower cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。