为大模型置信度信号设计可移植的验证协议，确保其可信可用。
Screen Before You Interpret: A Portable Validity Protocol for Benchmark-Based LLM Confidence Signals
- 基于临床心理评估原理，构建三指标+结构检验的验证框架
- 20个前沿模型中4个被判定无效，有效模型置信度与真实表现相关性达0.18
- 跨基准、跨格式验证通过，适用于安全决策等关键场景
大模型置信度信号用于拒答、路由和安全决策，但缺乏标准方法验证其是否具备个体项信息。本文借鉴临床人格评估（PAI、MMPI-3）的可验证性筛查原则，提出一种可移植的基准型置信度数据验证协议。该协议基于单个2×2列联表计算三个核心指数（L、Fp、RBS）、一个结构指标（TRIN）及项目敏感性统计量，采用四类临床传统建立三级分类体系（无效、不确定、有效）。在524个测试项上对20个前沿大模型进行验证，4个模型被判定为无效，2个为不确定。有效模型平均相关系数r = .18（15/16显著），无效模型平均r = -.20（d = 2.48）。在MMLU上使用语义化置信度对18个模型进行跨基准验证，并结合Yang等（2024）外部数据确认该筛查方法可跨基准与探测格式迁移。全部数据与代码公开于：https://github.com/synthiumjp/validity-scaling-llm
原文摘要 · Abstract (English)
LLM confidence signals are used for abstention, routing, and safety-critical decisions. No standard practice exists for checking whether a confidence signal carries item-level information before building on it. We transfer the validity screening principle from clinical personality assessment (PAI, MMPI-3) as a portable protocol for benchmark-based LLM confidence data. The protocol specifies three core indices (L, Fp, RBS), a structural indicator (TRIN), and an item-sensitivity statistic, computed from a single 2x2 contingency table. A three-tier classification system (Invalid, Indeterminate, Valid) draws on four clinical traditions. Validated on 20 frontier LLMs across 524 items, four models are classified Invalid, two Indeterminate. Valid-profile models show mean r = .18 (15/16 significant). Invalid-profile models show mean r = -.20 (d = 2.48). Cross-benchmark validation on 18 models using MMLU with verbalized confidence and on external data from Yang et al. (2024) confirms the screen transfers across benchmarks and probe formats. All data and code: https://github.com/synthiumjp/validity-scaling-llm
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。