通过块稀疏化提升大模型推理效率，降低能耗与成本
BLaST: High Performance Inference and Pretraining using BLock Sparse Transformers
- 迭代构建块稀疏结构，适配高效稀疏矩阵乘法
- 实现95%的MLP权重稀疏度，精度损失小于2.25%
- 支持所有线性层场景，适合大规模模型部署优化
大规模机器学习模型的能耗主要来自数据移动，需在内存层级和数据中心间传输数十亿参数。稀疏化通过剪枝冗余权重与激活值，有效减少数据移动。但现有方法常导致显著精度下降或性能开销。本文提出块稀疏变压器（BLaST），一种通用、鲁棒且可靠的稀疏化方法，适用于所有线性层场景。该方法迭代将权重矩阵稀疏化为适合高效稀疏矩阵-矩阵乘法（SpMM）的块稀疏模式。BLaST在MLP权重上实现最高95%的稀疏度，精度损失多数低于2.25%。在16个GPU上对Llama 3.2进行推理时，速度提升2.2倍；推理内存占用最多减少4.45倍，从而带来2.9倍的GPU部署与运行成本降低。
原文摘要 · Abstract (English)
The energy consumption of large-scale ML models is dominated by data movement, shuffling billions of parameters across memory hierarchies and data centers. Sparsification offers a principled way to mitigate these costs by pruning redundant weights and activations, thereby reducing data movement. Effective sparsification to prune redundant parameters is still challenging: existing methods incur significant accuracy degradation, performance overhead, or both. We introduce (Bl)ock (a)nd (S)parse (T)ransformers (BLaST), a general, robust, and reliable method for sparsification, applicable to linear layers in all settings. Our method iteratively sparsifies weight matrices into a block sparsity pattern suitable for efficient sparse matrix-matrix (SpMM) multiplication. BLaST achieves up to 95% sparsity in MLP weights with negligible accuracy loss (majority <2.25%). We show a 2.2x inference speedup for Llama 3.2 with 16 GPUs, and up to 4.45x reduction in inference memory footprint resulting in a 2.9x reduction in GPU setup and operating costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。