不训练不剪枝，用旋转激活实现稳定加速的LLM推理优化方法
La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
- 通过层间正交旋转转换激活值，提升稀疏化适配性
- 40%稀疏度下推理速度提升1.30倍，困惑度仅增0.17
- 无需训练即可通用，适合部署场景快速落地
激活稀疏性可降低大型语言模型推理时的计算开销和内存传输。现有方法或需耗时恢复训练阻碍实际应用，或依赖经验性的基于幅值剪枝，导致稀疏度波动和推理加速不稳定。本文提出LaRoSA（层间旋转稀疏激活），一种无需额外训练或幅值剪枝的激活稀疏化新方法。通过层间正交旋转将输入激活转化为更利于稀疏化的形式，在旋转后激活中采用Top-K选择，实现稳定的模型级稀疏性和可靠的时钟时间加速。LaRoSA在多种规模和类型的LLM上均有效，性能损失极小且推理加速稳健。具体而言，对于LLaMA2-7B在40%稀疏度下，困惑度仅增加0.17，时钟时间加速达1.30倍，零样本任务准确率差距缩小至0.54%，优于TEAL 1.77%，超越CATS 17.14%。
原文摘要 · Abstract (English)
Activation sparsity can reduce the computational overhead and memory transfers during the forward pass of Large Language Model (LLM) inference. Existing methods face limitations, either demanding time-consuming recovery training that hinders real-world adoption, or relying on empirical magnitude-based pruning, which causes fluctuating sparsity and unstable inference speed-up. This paper introduces LaRoSA (Layerwise Rotated Sparse Activation), a novel method for activation sparsification designed to improve LLM efficiency without requiring additional training or magnitude-based pruning. We leverage layerwise orthogonal rotations to transform input activations into rotated forms that are more suitable for sparsification. By employing a Top-K selection approach within the rotated activations, we achieve consistent model-level sparsity and reliable wall-clock time speed-up. LaRoSA is effective across various sizes and types of LLMs, demonstrating minimal performance degradation and robust inference acceleration. Specifically, for LLaMA2-7B at 40% sparsity, LaRoSA achieves a mere 0.17 perplexity gap with a consistent 1.30x wall-clock time speed-up, and reduces the accuracy gap in zero-shot tasks compared to the dense model to just 0.54%, while surpassing TEAL by 1.77% and CATS by 17.14%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。