用哈密顿蒙特卡洛优化神经网络最后层,提升不确定性估计效率。
Last Layer Hamiltonian Monte Carlo
- 仅对网络最后层进行哈密顿采样,降低计算开销。
- 在真实视频数据上表现良好,尤其提升异常检测能力。
- 适合资源有限但需可靠不确定性的实际部署场景。
本文探索将哈密顿蒙特卡洛(HMC)采样作为深度神经网络(DNN)的概率末层方法。尽管HMC被公认为不确定性估计的黄金标准,其计算成本限制了其在大规模数据和大型DNN中的应用。虽然采样参数的预测可并行化,但计算量仍随采样数线性增长(类似集成方法)。末层哈密顿蒙特卡洛(LL-HMC)通过仅对DNN最后一层进行采样,大幅减少计算量,使其适用于计算资源有限的数据密集型场景。本文在三个真实世界视频数据集上,对比了LL-HMC与五种其他末层概率深度学习方法,在驾驶员行为与意图识别任务中评估了分布内分类性能、校准度及分布外(OOD)检测能力。由于概率评估具有随机性,我们进行了五次网格搜索以避免依赖单一初始化。结果表明,LL-HMC在分布内分类和OOD检测上均达到竞争力水平。额外采样末层参数不提升分类性能,但有助于改善OOD检测;多链或不同起始点未带来一致改进。
原文摘要 · Abstract (English)
We explore the use of Hamiltonian Monte Carlo (HMC) sampling as a probabilistic last layer approach for deep neural networks (DNNs). While HMC is widely regarded as a gold standard for uncertainty estimation, the computational demands limit its application to large-scale datasets and large DNN architectures. Although the predictions from the sampled DNN parameters can be parallelized, the computational cost still scales linearly with the number of samples (similar to an ensemble). Last layer HMC (LL-HMC) reduces the required computations by restricting the HMC sampling to the final layer of a DNN, making it applicable to more data-intensive scenarios with limited computational resources. In this paper, we compare LL-HMC against five last layer probabilistic deep learning (LL-PDL) methods across three real-world video datasets for driver action and intention. We evaluate the in-distribution classification performance, calibration, and out-of-distribution (OOD) detection. Due to the stochastic nature of the probabilistic evaluations, we performed five grid searches for different random seeds to avoid being reliant on a single initialization for the hyperparameter configurations. The results show that LL-HMC achieves competitive in-distribution classification and OOD detection performance. Additional sampled last layer parameters do not improve the classification performance, but can improve the OOD detection. Multiple chains or starting positions did not yield consistent improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。