智能调度大模型评审团，按需调用不同专家并适时停止，提升评估效率。
Stopping and Routing LLM Judge Panels
- 根据任务条件动态分配评审角色，区分通用、互补与专业型评委
- 在多个评测场景中实现最高效率，相比全量调用节省超40%成本
- 适合需要高效、可审计的模型评估系统，尤其大型语言模型安全评测
大模型评估常面临众多候选评审：通用LLM评分提示、奖励模型、安全分类器、置信度变体及任务专用验证器。部署难题不仅在于选择最佳评审，更在于决定何时调用哪些评审、在哪些样本上使用以及何时终止评审组构建。本文将评审组设计建模为角色条件化分配问题。基于小规模标注审计集、声明的样本分片和评审成本，方法估计目标相对角色：冗余副本不增加条件信息，互补者提升整体面板表现，专业者仅在特定分片有效。由此推导出策略：剔除冗余副本，全局添加互补者，有条件路由专业者，并在验证增益低于阈值时停止。在推理、代码、安全、偏好、奖励模型、摘要与数学等多类审计中，该方法对比单个评审、扁平评审组、匹配多样性启发式、全量堆叠、可靠性评审团与节俭级联，结果形成评审调用的制度图谱：在可部署分片上路由专业者，饱和验证场景及时停止，当风险收益比合理则保留广泛集成，忽略条件副本。最终输出一份可复用、可审计的下一评估批次调用方案。
原文摘要 · Abstract (English)
LLM evaluation pipelines often have many candidate judges: general LLM-as-a-judge prompts, reward models, safety classifiers, confidence variants, and task-specific verifiers. The deployment question is not only which judge is best, but which judges should be called, on which examples, and when panel construction should stop. We formulate judge-panel design as a role-conditioned allocation problem. From a small labeled audit set, declared slices, and judge costs, the method estimates target-relative roles: copies add no conditional information, complements improve the global panel, and specialists help only on slices. These roles induce a policy: drop copies, add complements globally, route specialists conditionally, and stop when validation gain falls below a threshold. Across reasoning, code, safety, preference, reward-model, summarization, and math audits, the method is compared with single judges, flat panels, matched diversity heuristics, full-call stacking, reliability juries, and frugal cascades. The result is a regime map for judge calls: route specialists on deployable slices, stop in saturated verifier regimes, keep broad ensembles when their risk benefit is worth the cost, and ignore conditional copies. The output is a reusable, auditable call plan for the next evaluation batch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。