根据权重分布特性动态分配稀疏度，提升大模型推理速度与精度。
ActTail: Global Activation Sparsity in Large Language Models
- 基于重尾理论计算各投影层稀疏预算，实现全局自适应稀疏分配。
- 在80%稀疏度下，LLaMA-2-7B困惑度降低21.8%，性能优于均匀稀疏。
- 方法有理论支撑，适合追求高效高精度推理的模型部署场景。
激活稀疏性是加速大语言模型推理的有效手段，可减少计算量和内存移动。然而，现有方法通常对各投影层采用统一稀疏度，忽略了Transformer权重的异质统计特性，导致性能下降更严重。本文提出ActTail，一种基于TopK幅度的全局激活稀疏方法，其稀疏分配基于重尾自正则化（HT-SR）理论。具体而言，通过每个投影层的经验谱密度（ESD）计算重尾指数，作为量化指标来分配特定稀疏预算。我们进一步提供理论分析，建立激活稀疏率与重尾指数之间的显式关系，在HT-SR框架下为稀疏分配提供原则性指导，而非依赖启发式设计。在LLaMA与Mistral模型上的实验表明，本方法在高稀疏度下相比均匀分配显著提升困惑度与下游任务性能。在80%稀疏度下，LLaMA-2-7B的困惑度降低21.8%，LLaMA-2-13B降低40.1%，Mistral-7B降低9.4%。
原文摘要 · Abstract (English)
Activation sparsity is a promising approach for accelerating large language model (LLM) inference by reducing computation and memory movement. However, existing activation sparsity methods typically apply uniform sparsity across projections, ignoring the heterogeneous statistical properties of Transformer weights and thereby amplifying performance degradation. In this paper, we propose ActTail, a TopK magnitude-based activation sparsity method with global activation sparsity allocation grounded in Heavy-Tailed Self-Regularization (HT-SR) theory. Specifically, we capture this heterogeneity via the heavy-tail exponent computed from each projection's empirical spectral density (ESD), which is used as a quantitative indicator to assign projection-specific sparsity budgets. Importantly, we provide a theoretical analysis that establishes an explicit relationship between the activation sparsity ratio and the heavy-tail exponent under the HT-SR regime, offering principled guidance for sparsity allocation beyond heuristic design. Experiments on LLaMA and Mistral models show that our method improves both perplexity and downstream task performance at high sparsity compared to uniform allocation. At 80% sparsity, perplexity is reduced by 21.8% on LLaMA-2-7B, 40.1% on LLaMA-2-13B, and 9.4% on Mistral-7B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。