通过线性近似激活函数，实现大模型前馈网络80%参数压缩。
Accelerating Large Language Models through Partially Linear Feed-Forward Network
- 部分线性化非线性激活，高频输入用线性近似，异常输入动态回退。
- 在7B模型上实现80%参数压缩，精度比主流剪枝高65%。
- 适用于追求高效部署的模型，尤其适合vLLM和HuggingFace环境。
大型语言模型（LLMs）虽能力强大，但因参数量巨大难以部署。现有压缩技术如剪枝在高压缩比下会导致显著精度下降。受编译器常量折叠启发，本文提出新思路：将激活函数视为线性函数以减少参数。然而，现代LLM使用GELU等复杂非线性激活，无法直接应用。为此提出TARDIS，通过在高频输入范围内部分线性近似非线性激活，并对异常输入使用在线预测器动态回退至原始计算。实验表明，TARDIS在前馈网络中实现80%参数缩减，且相比Wanda和RIA等先进剪枝方法精度最高提升65%。在7B模型的实际部署中，结合vLLM系统可实现1.6倍端到端推理加速，使用HuggingFace时达1.4倍加速，仅损失10.9%精度。
原文摘要 · Abstract (English)
Large language models (LLMs) demonstrate remarkable capabilities but face deployment challenges due to their massive parameter counts. While existing compression techniques like pruning can reduce model size, it leads to significant accuracy degradation under high compression ratios. We present a novel perspective inspired by constant folding in compiler optimization. Our approach enables parameter reduction by treating activation functions in LLMs as linear functions. However, recent LLMs use complex non-linear activations like GELU that prevent direct application of this technique. We propose TARDIS, which enables optimization of LLMs with non-linear activations by partially approximating them with linear functions in frequently occurring input ranges. For outlier inputs, TARDIS employs an online predictor to dynamically fall back to original computations. Our experiments demonstrate that TARDIS achieves 80% parameter reduction in feed-forward networks, while significantly outperforming state-of-the-art pruning methods Wanda and RIA with up to 65% higher accuracy. In practical deployments for a 7B model, TARDIS achieves 1.6x end-to-end inference speedup when integrated with the vLLM serving system, and 1.4x speedup with the widely adopted HuggingFace implementation, while incurring only a 10.9% accuracy trade-off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。