8:16稀疏模式让大模型压缩更高效，尤其适合处理异常权重。
From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction
- 采用8:16半结构化稀疏，灵活度高于2:4且存储开销仅0.875比特/元素
- 在相同内存下，压缩模型性能超越未压缩模型，突破性能阈值
- 结合方差校正等技术，稀疏模型表现媲美甚至优于非结构化方法
随着大语言模型（LLMs）规模持续扩大，量化与稀疏化等高效压缩技术变得至关重要。虽然量化通过降低精度维持性能，但结构化稀疏方法（如N:M稀疏）因灵活性不足且对异常权重敏感而受限。本文探索8:16半结构化稀疏，证明其可在等效内存约束下使压缩模型性能超越未压缩或更小模型，达到性能阈值。相比2:4稀疏，8:16提供更高灵活性，存储开销仅为0.875比特/元素（2:4为0.75比特/元素）。此外，针对显著权重使用结构化稀疏模式，其效果可与非结构化方法比肩，甚至更优。最后，我们验证了方差校正、SmoothQuant类权重重均衡等简单技术能有效提升稀疏模型性能。
原文摘要 · Abstract (English)
As large language models (LLMs) grow in size, efficient compression techniques like quantization and sparsification are critical. While quantization maintains performance with reduced precision, structured sparsity methods, such as N:M sparsification, often fall short due to limited flexibility, and sensitivity to outlier weights. We explore 8:16 semi-structured sparsity, demonstrating its ability to surpass the Performance Threshold-where a compressed model matches the accuracy of its uncompressed or smaller counterpart under equivalent memory constraints. Compared to 2:4 sparsity, 8:16 offers greater flexibility with minimal storage overhead (0.875 vs. 0.75 bits/element). We also apply sparse structured patterns for salient weights, showing that structured sparsity for outliers is competitive with unstructured approaches leading to equivalent or better results. Finally, we demonstrate that simple techniques such as variance correction and SmoothQuant like weight equalization improve sparse models performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。