用剪枝MCMC训练紧凑贝叶斯神经网络,减少75%参数仍保持精度和不确定性估计。
Compact Bayesian Neural Networks via pruned MCMC sampling
- 结合MCMC采样与权重剪枝,移除冗余参数以压缩模型。
- 在多个数据集上实现超75%参数量减少,且性能损失可忽略。
- 适合需要轻量化、带不确定性的实际部署场景。
贝叶斯神经网络(BNN)能提供可靠的预测不确定性,但训练过程面临巨大计算挑战,主要源于使用马尔可夫链蒙特卡洛(MCMC)采样和变分推断算法时难以处理多模态后验分布。此外,模型参数数量随隐藏层、神经元及数据特征的增加呈指数级增长。通常大量连接参数冗余,剪枝不仅提升模型可移植性,还可能增强泛化能力。本研究通过融合MCMC采样与网络剪枝,构建紧凑的概率模型,移除低重要性权重。采样参数后验分布并剪枝低重要性权重,得到紧凑模型。通过后剪枝重采样确保模型保留不确定性估计能力及训练与泛化性能。在回归与分类基准数据集上验证策略有效性,并使用两个珊瑚礁岩芯岩性分类数据集测试其在复杂真实数据上的鲁棒性。进一步探究精炼紧凑BNN是否导致性能损失。结果表明,采用MCMC剪枝训练并压缩的BNN在超过75%参数量减少的情况下仍保持良好泛化性能,为可提供不确定性估计的紧凑型实际应用模型开辟了道路。
原文摘要 · Abstract (English)
Bayesian Neural Networks (BNNs) offer robust uncertainty quantification in model predictions, but training them presents a significant computational challenge. This is mainly due to the problem of sampling multimodal posterior distributions using Markov Chain Monte Carlo (MCMC) sampling and variational inference algorithms. Moreover, the number of model parameters scales exponentially with additional hidden layers, neurons, and features in the dataset. Typically, a significant portion of these densely connected parameters are redundant and pruning a neural network not only improves portability but also has the potential for better generalisation capabilities. In this study, we address some of the challenges by leveraging MCMC sampling with network pruning to obtain compact probabilistic models having removed redundant parameters. We sample the posterior distribution of model parameters (weights and biases) and prune weights with low importance, resulting in a compact model. We ensure that the compact BNN retains its ability to estimate uncertainty via the posterior distribution while retaining the model training and generalisation performance accuracy by adapting post-pruning resampling. We evaluate the effectiveness of our MCMC pruning strategy on selected benchmark datasets for regression and classification problems through empirical result analysis. We also consider two coral reef drill-core lithology classification datasets to test the robustness of the pruning model in complex real-world datasets. We further investigate if refining compact BNN can retain any loss of performance. Our results demonstrate the feasibility of training and pruning BNNs using MCMC whilst retaining generalisation performance with over 75% reduction in network size. This paves the way for developing compact BNN models that provide uncertainty estimates for real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。