无需训练即可加速大模型推理，提升90%激活稀疏性下的速度。
SVD Contextual Sparsity Predictors for Fast LLM Inference
- 用截断感知SVD构建无需训练的稀疏模式预测器
- 在90%激活稀疏度下实现1.8倍推理加速，性能损失<1%
- 支持CUDA与CANN设备，适合边缘部署场景
上下文稀疏性是降低大语言模型（LLM）推理计算复杂度的方法之一。现有基于上下文稀疏性的高效推理加速技术通常需要训练稀疏模式预测器。本文提出一种加速基于ReGLU的前馈网络（FFN）推理的框架，采用截断感知奇异值分解（SVD）对门控投影矩阵进行处理，并结合阈值校准算法，构建无需训练的稀疏模式预测器，同时提供支持条件计算的推理执行器，可在CUDA和CANN设备上运行。在三个稀疏大模型上的实验表明，其FFN层平均激活稀疏度达90%，端到端解码时间最高减少1.8倍，且在涉及复杂数学和代码生成的任务中，基准得分下降不足1%。该工作推动了大模型在边缘设备上的部署。
原文摘要 · Abstract (English)
Contextual sparsity is one of the approaches used to reduce computational complexity in the inference process of large language models (LLMs). Existing techniques for efficient LLM inference acceleration based on contextual sparsity with minimal accuracy degradation require training sparse pattern predictors. This paper presents a framework for accelerating inference of ReGLU-based feed-forward networks (FFNs) within LLMs. The proposed framework provides a fast, training-free method for building sparse pattern predictors using truncation-aware singular value decomposition (SVD) of the gate projection matrix, along with a threshold calibration algorithm, and inference executors supporting conditional computation on CUDA and CANN devices. Experiments on three sparse LLMs with an average activation sparsity level of 90% in the FFNs demonstrate up to a 1.8x reduction in end-to-end decoding time while maintaining less than 1% degradation in benchmark scores on tasks involving complex math and code generation. This work advances the deployment of LLMs on edge devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。