MACKO提升低稀疏度下大模型推理效率，实现内存减半、速度翻倍。
MACKO: Sparse Matrix-Vector Multiplication for Low Sparsity
- 设计新型稀疏矩阵格式与GPU内核协同优化，适配非结构化稀疏场景。
- 在50%稀疏度下，相比密集表示内存减少1.5倍、速度提升1.2-1.5倍。
- 无需专用硬件，可直接用于真实大模型推理，适合追求高效部署的开发者。
稀疏矩阵-向量乘法（SpMV）是稀疏大语言模型（LLMs）推理中的基础操作。由于现有SpMV方法在剪枝后常见的低且非结构化稀疏度（30%-90%）下表现不佳，非结构化剪枝仅带来有限的内存压缩和加速。我们提出MACKO-SpMV，一种为GPU优化的格式与内核协同设计方法，在降低存储开销的同时保持对GPU执行模型的兼容性。该方法无需专用硬件单元（如张量核心）或格式特定预计算，即可高效支持非结构化稀疏。实测结果表明，在50%稀疏度下，MACKO相较密集表示实现1.5倍内存减少和1.2-1.5倍加速；相比其他基线：比cuSPARSE快2.8-13.0倍，比Sputnik快1.9-2.6倍，比DASP快2.2-2.5倍。应用于以Wanda剪枝至50%稀疏度的Llama2-7B模型时，可在fp16精度下实现1.5倍内存减少和1.5倍推理加速。得益于MACKO，50%稀疏度的非结构化剪枝现在在真实大模型工作负载中具有实际意义。
原文摘要 · Abstract (English)
Sparse Matrix-Vector Multiplication (SpMV) is a fundamental operation in the inference of sparse Large Language Models (LLMs). Because existing SpMV methods perform poorly under the low and unstructured sparsity (30-90%) commonly observed in pruned LLMs, unstructured pruning provided only limited memory reduction and speedup. We propose MACKO-SpMV, a GPU-optimized format and kernel co-designed to reduce storage overhead while preserving compatibility with the GPU's execution model. This enables efficient SpMV for unstructured sparsity without specialized hardware units (e.g., tensor cores) or format-specific precomputation. Empirical results show that at sparsity 50%, MACKO is the first approach with significant 1.5x memory reduction and 1.2-1.5x speedup over dense representation. Speedups over other SpMV baselines: 2.8-13.0x over cuSPARSE, 1.9-2.6x over Sputnik, and 2.2-2.5x over DASP. Applied to Llama2-7B pruned with Wanda to sparsity 50%, it delivers 1.5x memory reduction and 1.5x faster inference at fp16 precision. Thanks to MACKO, unstructured pruning at 50% sparsity is now justified in real-world LLM workloads.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。