用信息瓶颈理论提升模型对异常样本的检测能力,效果远超传统方法。
VNDUQE: Information-Theoretic Novelty Detection using Deep Variational Information Bottleneck

- 基于信息瓶颈原理构建深度神经网络,约束特征表示中的信息流动。
- 在噪声和新数字类别上分别达到100%和94.7%的检测准确率(AUROC)。
- 适合需要高可靠性异常检测的工业检测与主动学习场景。
在安全关键应用中,检测分布外(OOD)样本对神经网络的安全部署至关重要。尽管最大软分数(MSP)提供简单基线,但缺乏理论基础且存在校准偏差。本文提出基于变分信息瓶颈(VIB)的新型检测方法VNDUQE,通过约束学习表征中的信息流实现新颖性检测。在保留部分数字类别的MNIST数据集上训练模型,并使用信息论指标(KL散度与预测熵)评估分布外检测性能。结果表明两种信号互补:KL散度在远距离分布外样本(噪声、域偏移)上达到100% AUROC;预测熵在近似分布外样本(新数字类别)上表现优异,达94.7% AUROC。结合两者可实现95.3%平均AUROC,5%假阳性率下92%真阳性率,较基线MSP提升32个百分点(85.0% AUROC,60.1% TPR)。通过信息瓶颈压缩(β=10⁻³)使期望校准误差降低38%,证明信息论约束能生成更可靠的不确定性估计。该成果支持昂贵计算资源下的主动学习,使阈值设定更具合理性。
原文摘要 · Abstract (English)
Detecting out-of-distribution (OOD) samples is critical for safe deployment of neural networks in safety-critical applications. While maximum softmax probability (MSP) provides a simple baseline, it lacks theoretical grounding and suffers from miscalibration. We propose VNDUQE (VIB-based Novelty Detection and Uncertainty Quantification for Nondestructive Evaluation), which investigates novelty detection through the Deep Variational Information Bottleneck (VIB), which explicitly constrains information flow through learned representations. We train VIB models on MNIST with held-out digit classes and evaluate OOD detection using information-theoretic metrics: KL divergence and prediction entropy. Our results reveal complementary detection signals: KL divergence achieves perfect detection (100\% AUROC on noise) on far-OOD samples (noise, domain shift), while prediction entropy excels at near-OOD detection (94.7\% AUROC on novel digit classes). A parallel detection strategy combining both metrics achieves 95.3\% average AUROC and 92\% true positive rate at 5\% false positive rate, which is a 32 percentage point improvement over baseline MSP (85.0\% AUROC, 60.1\% TPR). Compression via the information bottleneck principle ($β=10^{-3}$) reduces Expected Calibration Error by 38\%, demonstrating that information-theoretic constraints produce fundamentally more reliable uncertainty estimates. These findings directly support active learning with expensive computational oracles, where well-calibrated novelty detection enables principled threshold selection for oracle queries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。