通过线性组合发现并过滤冗余权重，大幅降低大模型推理计算量。
COUNTDOWN: Contextually Sparse Activation Filtering Out Unnecessary Weights in Down Projection
- 基于下投影矩阵的线性组合特性，动态筛选非必要参数。
- D-COUNTDOWN可省90%计算，性能损失仅5.5%；M-COUNTDOWN无需预测器且保真度更高。
- 专有内核实现理论优势，在实际中显著提速，适合部署优化场景。
大型语言模型规模增长导致严重的计算效率问题。为应对这一挑战，稀疏激活方法在推理过程中选择性禁用非必要参数，以降低前馈神经网络（FFNN）层的计算开销。现有方法多依赖非线性门控机制，我们提出新假设：FFNN层的稀疏性在全局上表现为内部下投影矩阵的线性组合形式。基于此，提出两种方法：M-COUNTDOWN利用间接系数，D-COUNTDOWN利用直接系数。实验表明，D-COUNTDOWN在理想情况下可省去90%计算，性能损失低至5.5%；而M-COUNTDOWN提供无预测器方案，相比现有方法性能保留提升高达29.4%。专用内核实现有效将理论收益转化为实际加速。
原文摘要 · Abstract (English)
The growing size of large language models has created significant computational inefficiencies. To address this challenge, sparse activation methods selectively deactivates non-essential parameters during inference, reducing computational costs in FFNN layers. While existing methods focus on non-linear gating mechanisms, we hypothesize that the sparsity of the FFNN layer lies globally in the form of a linear combination over its internal down projection matrix. Based on this insight, we propose two methods: M-COUNTDOWN, leveraging indirect coefficients, and D-COUNTDOWN, utilizing direct coefficients of the linear combination. Experimental results demonstrate that D-COUNTDOWN can omit 90% of computations with performance loss as low as 5.5% ideally, while M-COUNTDOWN provides a predictor-free solution with up to 29.4% better performance preservation compared to existing methods. Our specialized kernel implementations effectively realize these theoretical gains into substantial real-world acceleration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。