用程序代替大模型做评分，又快又透明还省钱。
Codifying the Judge: Scalable Evaluation via Program Distillation
- 把大模型的评分逻辑转成可读可改的程序集合
- 在5个数据集上达到130亿参数大模型的评分水平
- 适合需要高效、可靠评估的AI研发团队
大模型作为评判者已成为自动化评估的标准，但存在成本高、延迟大、决策不透明等问题，制约其可扩展性和可靠性。本文提出一种简单高效的替代方案：程序蒸馏。不再在评估时调用大模型，而是将其决策逻辑蒸馏为一组程序，直接对候选内容进行打分。这些程序化评判者具备透明性，易于检查和修改，并消除每次样本的API费用。基于此，我们提出PAJAMA系统，该系统合成程序作为评判者，聚合其判断结果形成综合结论，并引入回退机制，在置信度低时自动将任务转交大模型处理。在五个数据集和四个模型族上，程序化评判者的性能可媲美130亿参数的大模型评判者。当以程序输出作为路由信号时，PAJAMA同时提升准确率与吞吐量，推动了性能-效率权衡的前沿。此外，程序化评判者还能生成廉价高效的奖励信号：在RewardBench上，由程序判断结果训练的奖励模型，在仅需大模型标签成本百分之一的情况下，表现优于基于专有大模型标签训练的模型。
原文摘要 · Abstract (English)
LLM-as-a-judge has become the standard for automated evaluation, but it suffers from high cost, significant latency, and opaque decisions -- limitations that undermine its scalability and reliability. We address these with a simple, efficient alternative: program distillation. Instead of prompting an LLM at the evaluation time, we distill its decision logic into a committee of programs that score candidates directly. These programmatic judges offer transparency, are easily inspected or edited, and eliminate per-sample API costs. Building on this notion, we introduce PAJAMA, a system that synthesizes programs as judges, aggregates their decisions into a joint verdict, and incorporates a fallback mechanism to selectively escalate low-confidence cases to an LLM. Across five datasets and four model families, we show that programmatic judges can match the performance of a 13B-size LLM judge. When using program outputs as routing signals, PAJAMA improves both accuracy and throughput and advances the Pareto frontier. Beyond evaluation, programmatic judges produce cheap and effective reward signals: on RewardBench, a reward model distilled from programs' verdicts outperforms one trained on a proprietary LLM's labels at two orders of magnitude lower API cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。