通过对比逻辑值一致性,揭示不同模型对不确定性的编码差异。
Bag of Coins: A Statistical Probe into Neural Confidence Structures
- 用配对主导概率聚合检验逻辑值一致性,生成确定性得分与统计显著性评分。
- ViT在分布内/外样本间有明显分离(差距0.1-0.6),而ResNet和RoBERTa重叠严重。
- 适合作为研究工具分析模型内部不确定性结构,不推荐用于实际校准或异常检测。
现代神经网络常出现置信度校准偏差且难以识别分布外输入,现有方法多为输出后处理,缺乏对内部一致性的检验。本文提出非参数诊断工具Bag-of-Coins(BoC),通过比较Softmax置信度$ ilde{p}$与成对Luce风格主导概率的聚合$ar{q}$,获得确定性一致性得分与基于p值的结构得分。在ViT、ResNet和RoBERTa模型上测试,一致性差距$Δ = ar{q} - ilde{p}$显示:ViT在分布内(约0.1-0.2)与分布外(约0.5-0.6)有明显分离,而ResNet和RoBERTa均接近0,表明不确定性几何结构受架构影响。作为实用方法,仅当基础模型校准较差时(如ViT:ECE从0.180降至0.024),BoC才有效,但仍劣于标准校准器(ECE~0.005);在分布外检测中,其表现远低于标准方法(AUROC 0.020-0.253 vs. 0.75-0.99)。因此,我们主张将BoC定位为研究诊断工具,用于探究架构如何在逻辑值空间编码不确定性。
原文摘要 · Abstract (English)
Modern neural networks often produce miscalibrated confidence scores and struggle to detect out-of-distribution (OOD) inputs, while most existing methods post-process outputs without testing internal consistency. We introduce the Bag-of-Coins (BoC) probe, a non-parametric diagnostic of logit coherence that compares softmax confidence $\hat p$ to an aggregate of pairwise Luce-style dominance probabilities $\bar q$, yielding a deterministic coherence score and a p-value-based structural score. Across ViT, ResNet, and RoBERTa with ID/OOD test sets, the coherence gap $Δ=\bar q-\hat p$ reveals clear ID/OOD separation for ViT (ID ${\sim}0.1$-$0.2$, OOD ${\sim}0.5$-$0.6$) but substantial overlap for ResNet and RoBERTa (both ${\sim}0$), indicating architecture-dependent uncertainty geometry. As a practical method, BoC improves calibration only when the base model is poorly calibrated (ViT: ECE $0.024$ vs.\ $0.180$) and underperforms standard calibrators (ECE ${\sim}0.005$), while for OOD detection it fails across architectures (AUROC $0.020$-$0.253$) compared to standard scores ($0.75$-$0.99$). We position BoC as a research diagnostic for interrogating how architectures encode uncertainty in logit geometry rather than a production calibration or OOD detection method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。