通过贝叶斯变分学习实现权重稀疏化与低比特量化统一压缩。
SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions
- 用稀疏板-泥浆先验和高斯混合模型建模权重分布。
- 在多个模型上实现更高压缩率，性能损失相近。
- 适合资源受限设备部署的深度模型压缩场景。
大规模神经网络的压缩对在资源受限设备上部署模型至关重要。现有方法多单独采用权重剪枝或低比特量化，常导致压缩率不足而性能下降。本文提出一种基于贝叶斯变分学习的统一框架（SQS），实现剪枝与低比特量化的协同优化，在保持相当性能的前提下获得更高压缩率。核心思想是引入稀疏板-泥浆先验以诱导稀疏性，并使用高斯混合模型（GMM）对量化权重进行建模，支持低比特精度。理论上，我们证明了该变分方法可一致逼近稀疏且量化的深度神经网络。在ResNet、BERT-base、Llama3和Qwen2.5等模型上的大量实验表明，本方法相比现有基线实现了更高的压缩率，且性能损失相当。
原文摘要 · Abstract (English)
Compressing large-scale neural networks is essential for deploying models on resource-constrained devices. Most existing methods adopt weight pruning or low-bit quantization individually, often resulting in suboptimal compression rates to preserve acceptable performance drops. We introduce a unified framework for simultaneous pruning and low-bit quantization via Bayesian variational learning (SQS), which achieves higher compression rates than prior baselines while maintaining comparable performance. The key idea is to employ a spike-and-slab prior to inducing sparsity and model quantized weights using Gaussian Mixture Models (GMMs) to enable low-bit precision. In theory, we provide the consistent result of our proposed variational approach to a sparse and quantized deep neural network. Extensive experiments on compressing ResNet, BERT-base, Llama3, and Qwen2.5 models show that our method achieves higher compression rates than a line of existing methods with comparable performance drops.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。