无需额外调用，从大模型推理过程提取可信度，解决自动评分系统信任难题。
VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference
- 分解验证步骤，从推理轨迹中提取三项结构化信号构建置信度
- 在多个模型上实现0.66-0.91的AUROC，尤其在日志概率失效时仍有效
- 适合需高可信自动评估的工业级系统，尤其对闭源模型友好
LLM作为评判者广泛用于自动化评估，但从业者缺乏判断其结论可信度的方法。传统的基于词元概率的置信度信号在许多商用大模型中不可用，且在结构化输出下超过0.999即饱和。我们提出VERDI（VERification-Decomposed Inference），从已有结构化评判的推理轨迹中提取置信度，无需额外推理调用。VERDI将每项验证式评估拆分为子检查，生成三个结构性信号：步骤-结论一致性、主张级置信差值、证据锚定得分，并通过Platt校准的逻辑回归融合。在三个公开基准测试中，VERDI在GPT-4.1-mini上达到0.72-0.91的AUROC，GPT-5.4-mini上为0.66-0.80。在Qwen3.5-4B/9B/27B上，由于答案词元概率反校准（错误时信心更高，AUROC 0.32-0.49），VERDI仍达到0.56-0.70。我们还在生产系统中验证了八个评分维度（事实类评分的AUROC为0.73-0.88），展示了跨模型迁移能力（AUROC 0.66-0.69），并证明一个3300万参数的NLI模型可作为可扩展的正则表达式替代方案。
原文摘要 · Abstract (English)
LLM-as-Judge systems are widely deployed for automated evaluation, yet practitioners lack reliable methods to know when a judge's verdict should be trusted. Token log-probabilities, the standard post-hoc confidence signal, are unavailable for many commercial LLMs and, even when accessible, saturate above 0.999 with structured JSON output. We introduce VERDI (VERification-Decomposed Inference), a method that extracts confidence from the reasoning trace a structured judge already produces, with no additional inference calls. VERDI decomposes each verification-style evaluation into sub-checks and derives three structural signals: Step-Verdict Alignment, Claim-Level Margin, and Evidence Grounding Score. We combine them with Platt-scaled logistic regression. On three public benchmarks, VERDI achieves AUROC 0.72-0.91 on GPT-4.1-mini and 0.66-0.80 on GPT-5.4-mini. On Qwen3.5-4B/9B/27B, where answer-token logprobs are anti-calibrated (higher confidence on errors, AUROC 0.32-0.49), VERDI achieves 0.56-0.70. We additionally validate on a production system with eight rubrics (AUROC 0.73-0.88 on factual rubrics), demonstrate cross-model transfer (AUROC 0.66-0.69), and show that a 33M-parameter NLI (Natural Language Inference) model provides a scalable alternative to regex extraction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。