用线性近似加速大模型推理，速度提升32%且精度损失低于1%。
Efficient Large Language Model Inference with Neural Block Linearization
- 用线性最小均方误差估计替代自注意力层，实现高效推理
- 在12个层上应用后推理提速32%，准确率下降不足1%
- 无需微调即可部署，适合追求效率的模型落地场景
基于Transformer的大语言模型（LLM）推理需求高，部署困难。本文提出神经块线性化（NBL）框架，通过线性最小均方误差估计器对自注意力层进行线性近似，以加速推理。NBL利用典型相关分析计算近似误差的理论上限，并以此为标准选择线性化误差最低的模型层。该方法可直接应用于预训练的LLM，无需微调。实验表明，将NBL应用于DeepSeek-R1-Distill-Llama-8B的12个自注意力层，推理速度提升32%，准确率下降不足1%，在多个推理基准上保持竞争力。代码已开源：https://github.com/LIONS-EPFL/NBL。
原文摘要 · Abstract (English)
The high inference demands of transformer-based Large Language Models (LLMs) pose substantial challenges in their deployment. To this end, we introduce Neural Block Linearization (NBL), a novel framework for accelerating transformer model inference by replacing self-attention layers with linear approximations derived from Linear Minimum Mean Squared Error estimators. NBL leverages Canonical Correlation Analysis to compute a theoretical upper bound on the approximation error. Then, we use this bound as a criterion for substitution, selecting the LLM layers with the lowest linearization error. NBL can be efficiently applied to pre-trained LLMs without the need for fine-tuning. In experiments, NBL achieves notable computational speed-ups while preserving competitive accuracy on multiple reasoning benchmarks. For instance, applying NBL to 12 self-attention layers in DeepSeek-R1-Distill-Llama-8B increases the inference speed by 32% with less than 1% accuracy trade-off, making it a flexible and promising solution to improve the inference efficiency of LLMs. The implementation is available at: https://github.com/LIONS-EPFL/NBL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。