用软阈值自动压缩语言模型，提速超1.3倍且参数减半。
SoftLMs: Efficient Adaptive Low-Rank Approximation of Language Models using Soft-Thresholding Mechanism
- 通过可微软阈值动态确定每层秩，自动实现最优压缩。
- 在BERT/GPT2/TinyLlama上实现1.33~1.72倍加速，参数减少50%。
- 适用于Transformer与Mamba等架构，适合资源受限部署。
为提升语言模型性能，研究者广泛采用基于注意力的Transformer结构。然而，高维线性层带来显著的计算与内存开销。随着模型计算需求上升，亟需压缩技术以实现在资源受限设备上的部署。本文提出一种新型压缩方法，利用软阈值机制动态决定每层秩，以可微方式裁剪小幅值奇异值。该方法自动化决策过程，识别各层最优压缩程度。我们已将该技术应用于BERT（判别任务）及GPT2、TinyLlama（生成任务）等注意力架构，并在最近提出的状态空间模型Mamba上验证。实验表明，该方法在编码器/解码器中实现1.33倍至1.72倍加速，总参数减少50%。
原文摘要 · Abstract (English)
Extensive efforts have been made to boost the performance in the domain of language models by introducing various attention-based transformers. However, the inclusion of linear layers with large dimensions contributes to significant computational and memory overheads. The escalating computational demands of these models necessitate the development of various compression techniques to ensure their deployment on devices, particularly in resource-constrained environments. In this paper, we propose a novel compression methodology that dynamically determines the rank of each layer using a soft thresholding mechanism, which clips the singular values with a small magnitude in a differentiable form. This approach automates the decision-making process to identify the optimal degree of compression for each layer. We have successfully applied the proposed technique to attention-based architectures, including BERT for discriminative tasks and GPT2 and TinyLlama for generative tasks. Additionally, we have validated our method on Mamba, a recently proposed state-space model. Our experiments demonstrate that the proposed technique achieves a speed-up of 1.33X to 1.72X in the encoder/ decoder with a 50% reduction in total parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。