用双稀疏分解让神经网络更小更强，13B模型压缩50%还胜过7B原版。
Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorization
- 将权重矩阵拆成两个稀疏矩阵，实现高效压缩。
- 130亿参数模型压缩50%后性能超70亿参数原版。
- 适合追求极致模型压缩与推理加速的开发者。
神经网络因规模庞大而难以部署。本文提出双稀疏分解（DSF），将每个权重矩阵分解为两个稀疏矩阵。尽管精确求解不可行，我们设计基于ADMM的交替最小化启发式算法，实现当前最优压缩效果。例如，在一次性剪枝设置下，可将LLaMA2-13B模型大小减少50%，性能优于密集的LLaMA2-7B模型。该方法在卷积神经网络上也优于最优大脑压缩（Optimal Brain Compression）等层级剪枝方法。且经微调后性能仍保持优势。代码已开源：https://github.com/usamec/double_sparse。
原文摘要 · Abstract (English)
Neural networks are often challenging to work with due to their large size and complexity. To address this, various methods aim to reduce model size by sparsifying or decomposing weight matrices, such as magnitude pruning and low-rank or block-diagonal factorization. In this work, we present Double Sparse Factorization (DSF), where we factorize each weight matrix into two sparse matrices. Although solving this problem exactly is computationally infeasible, we propose an efficient heuristic based on alternating minimization via ADMM that achieves state-of-the-art results, enabling unprecedented sparsification of neural networks. For instance, in a one-shot pruning setting, our method can reduce the size of the LLaMA2-13B model by 50% while maintaining better performance than the dense LLaMA2-7B model. We also compare favorably with Optimal Brain Compression, the state-of-the-art layer-wise pruning approach for convolutional neural networks. Furthermore, accuracy improvements of our method persist even after further model fine-tuning. Code available at: https://github.com/usamec/double_sparse.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。