通过低秩权重共享，实现大模型高效剪枝与扩展。
FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharing
- 按重要性剪枝模块，用低秩适配器和权重共享替换。
- 30%压缩率下5/6基准达顶尖性能，40%压缩率全胜。
- 可低成本扩展小模型，仅需0.3%训练tokens提升性能。
大型语言模型（LLM）在自然语言处理中的快速普及，对内存受限设备上的高效部署提出了迫切需求。本文提出一种方法，根据重要性评分选择性剪枝模型模块，并采用低参数替代策略进行替换。具体而言，我们设计了一种原则性度量标准，利用未剪枝模块的对应特征和模块特定的低秩适配器，通过权重共享机制替换每个被剪枝的块。此外，我们通过输出特征归一化和基于低秩SVD重构的适配器初始化方案，促进替代块的学习。实证评估表明，该方法在多个基准上显著优于现有技术：在30%压缩率下，5/6基准达到当前最优；在40%压缩率下，6/6基准均表现领先。同时，我们的方法还能扩展小型模型，在仅使用约0.3%扩展训练令牌且附加参数极少的情况下，6/6基准性能均得到提升。
原文摘要 · Abstract (English)
The rapid proliferation of large language models (LLMs) in natural language processing (NLP) has created a critical need for techniques that enable efficient deployment on memory-constrained devices without compromising performance. We present a method to prune LLMs that selectively prunes model blocks based on an importance score and replaces them with a low-parameter replacement strategy. Specifically, we propose a principled metric to replace each pruned block using a weight-sharing mechanism that leverages unpruned counterparts from the model and block-specific low-rank adapters. Furthermore, we facilitate the learning of these replacement blocks with output feature normalization and an adapter initialization scheme built on low-rank SVD reconstructions. Empirical evaluations demonstrate substantial performance gains over existing methods, achieving state-of-the-art performance on 5/6 benchmarks for a compression rate of 30% and 6/6 benchmarks for a compression rate of 40%. We also demonstrate that our approach can extend smaller models, boosting performance on 6/6 benchmarks using only ~0.3% tokens of extended training with minimal additional parameter costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。