用随机权重共享技术让贝叶斯神经网络训练更快更省
Stochastic Weight Sharing for Bayesian Neural Networks
- 将量化技术从随机视角重构，用低维高斯表示模拟贝叶斯网络行为
- 在多个视觉数据集上实现50倍参数压缩、75%模型体积减小
- 适合需要高效不确定性估计的大模型场景
尽管贝叶斯神经网络（BNNs）为深度学习中的不确定性量化提供了理论框架，但其在训练超深、前沿架构时仍受限于计算开销大和收敛困难。本文从随机角度重新诠释权重共享量化技术，引入二维自适应高斯分布、Wasserstein距离估计与alpha混合，以低维软高斯表示编码BNN的随机特性。通过大量实验验证，该方法使贝叶斯学习的计算开销降低数个数量级，可高效训练大规模模型如ResNet-101和Vision Transformer（VIT）。在CIFAR10、CIFAR100和ImageNet1k等基准上，模型参数压缩约50倍，模型尺寸减少75%，同时保持与顶尖水平相当的准确率和不确定性估计性能。
原文摘要 · Abstract (English)
While offering a principled framework for uncertainty quantification in deep learning, the employment of Bayesian Neural Networks (BNNs) is still constrained by their increased computational requirements and the convergence difficulties when training very deep, state-of-the-art architectures. In this work, we reinterpret weight-sharing quantization techniques from a stochastic perspective in the context of training and inference with Bayesian Neural Networks (BNNs). Specifically, we leverage 2D adaptive Gaussian distributions, Wasserstein distance estimations, and alpha blending to encode the stochastic behaviour of a BNN in a lower dimensional, soft Gaussian representation. Through extensive empirical investigation, we demonstrate that our approach significantly reduces the computational overhead inherent in Bayesian learning by several orders of magnitude, enabling the efficient Bayesian training of large-scale models, such as ResNet-101 and Vision Transformer (VIT). On various computer vision benchmarks including CIFAR10, CIFAR100, and ImageNet1k. Our approach compresses model parameters by approximately 50x and reduces model size by 75, while achieving accuracy and uncertainty estimations comparable to the state-of-the-art.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。