提出高效无采样不确定性估计方法,提升大模型可靠性。
Calibrated Sampling-Free Uncertainty Estimation in Bayesian Deep Learning
- 基于方差传播构建单次前向的不确定性计算框架
- 在视觉推理任务中覆盖率达14.6%,显著优于之前方法
- 适用于Transformer与CNN,适合高风险场景部署
现代深度学习模型普遍存在过度自信问题,限制其在高风险应用中的可靠性。贝叶斯方法通过学习参数分布来缓解此问题,近年已可低成本应用于大规模架构。但测试时需多次采样权重平均预测,成本过高。方差传播可在单次前向中快速估算不确定性,但难以扩展至深层多样结构。本文提出校准方差传播(CVP),针对归一化层设计新传播机制,结合激活函数处理技术,并通过轻量校准消除残差误差。CVP在Transformer与CNN上实现与蒙特卡洛采样相当的不确定性估计精度,代价仅为后者的极小部分。相比先前方法,CVP在BEiT-3上的视觉推理(NLVR2)任务中,0.5%风险下的覆盖率从8.2%提升至14.6%;在ViLT的VQAv2任务中,从2.6%提升至10.8%,并推广至卷积网络。
原文摘要 · Abstract (English)
Modern deep learning models remain notoriously prone to overconfidence, limiting their reliability in high-stakes applications. Bayesian methods aim to counter this by learning a distribution over model parameters, and recent advances now make this feasible for large-scale architectures at costs comparable to AdamW. However, a challenge remains at test time: predictions must be averaged across many forward passes with weights sampled from the posterior, which is prohibitively expensive. Variance propagation offers an efficient alternative, computing layer-wise analytical approximations of uncertainty in a single forward pass. While such techniques are effective for MLPs, their extension to modern architectures remains challenging, due to increased depth and diversity of layer types. To fill this gap, we propose Calibrated Variance Propagation (CVP), which introduces a new propagation method for normalization layers, combines it with recent techniques for handling activation functions, and absorbs residual error through a light calibration step. CVP yields comparably accurate uncertainty estimates to MC sampling across transformers and CNNs, at a fraction of the cost. Against prior variance propagation work, CVP improves coverage at $0.5\%$ risk from $8.2\%$ to $14.6\%$ with BEiT-3 on Visual Reasoning (NLVR2) and from $2.6\%$ to $10.8\%$ with ViLT on VQAv2, with gains extending to convolutional architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。