通过稀疏化前馈层，让大模型更快更省电。
Sparser, Faster, Lighter Transformer Language Models
- 用L1正则诱导99%稀疏度，只损失少量性能。
- 稀疏计算提升推理与训练吞吐，规模越大越省能。
- 开源稀疏核与格式，助力高效模型落地。
自回归大语言模型（LLM）的扩展带来了前所未有的进展，但也伴随着巨大的计算开销。本文针对这一问题，通过在LLM的前馈层中引入非结构化稀疏性来降低计算成本，这些层占用了大部分参数和执行浮点运算量（FLOPs）。为此，我们提出了一种新的稀疏打包格式和一系列专为现代GPU优化的CUDA内核，可无缝集成到现有执行流水线中，实现高效的稀疏计算。我们还进行了定量研究，表明简单的L1正则化即可诱导超过99%的稀疏度，对下游任务性能影响微乎其微。结合我们的内核，在不同规模下均显著提升了吞吐量、能效比和内存占用。所有代码和内核将开源，以推动稀疏性成为提升基础模型效率与可扩展性的实用路径。
原文摘要 · Abstract (English)
Scaling autoregressive large language models (LLMs) has driven unprecedented progress but comes with vast computational costs. In this work, we tackle these costs by leveraging unstructured sparsity within an LLM's feedforward layers, the components accounting for most of the model parameters and execution FLOPs. To achieve this, we introduce a new sparse packing format and a set of CUDA kernels designed to seamlessly integrate with the optimized execution pipelines of modern GPUs, enabling efficient sparse computation during LLM inference and training. To substantiate our gains, we provide a quantitative study of LLM sparsity, demonstrating that simple L1 regularization can induce over 99% sparsity with negligible impact on downstream performance. When paired with our kernels, we show that these sparsity levels translate into substantial throughput, energy efficiency, and memory usage benefits that increase with model scale. We will release all code and kernels under an open-source license to promote adoption and accelerate research toward establishing sparsity as a practical axis for improving the efficiency and scalability of modern foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。