用相似性指导分层剪枝,高效压缩大模型且保持性能
SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Models
- 基于表示相似性分组网络层,确保剪枝时保留语义关联
- 在图像分类和大语言模型上实现更高精度与更强压缩比
- 无需微调即可评估层重要性,适合资源受限场景部署
层剪枝已成为压缩预训练网络、降低模型规模并提升计算效率的有效方法。然而,现有方法大多忽视了深层神经网络中各层之间的内在联系与相互依赖关系,导致剪枝后模型未能充分保留原始网络的关键特征。为此,本文提出一种新颖的相似性引导分层剪枝框架(SGLP),利用表示相似性指导高效且有依据的层移除。首先通过中心核对齐(CKA)量化层间表示相似性,揭示网络内部结构模式;随后对相似性矩阵应用费雪最优分割(Fisher Optimal Segmentation),将网络划分为语义一致的层段;在此基础上,每段内使用无需微调的GradNorm评估层重要性,实现目标化、分段式冗余层移除。在图像分类任务及大语言模型上的实验表明,SGLP在精度与效率上均优于当前最先进方法,实现显著模型压缩且性能损失极小,适用于资源受限环境部署。
原文摘要 · Abstract (English)
Layer pruning has emerged as a potent approach to remove redundant layers in the pre-trained network on the purpose of reducing network size and improve computational efficiency. However, existing layer pruning methods mostly overlook the intrinsic connections and inter-dependencies between different layers within complicated deep neural networks. This oversight can result in pruned models that do not preserve the essential characteristics of the pre-trained network as effectively as desired. To address these limitations, we propose a Similarity-Guided Layer Partition (SGLP) Pruning, a novel pruning framework that exploits representation similarity to guide efficient and informed layer removal for compressing large deep models. Our method begins by employing Centered Kernel Alignment (CKA) to quantify representational similarity between layers, uncovering structural patterns within the network. We then apply Fisher Optimal Segmentation on the similarity matrix to partition the network into semantically coherent layer segments. This segmentation allows pruning decisions to respect layer interdependencies and preserve essential knowledge. Within each segment, we introduce a fine-tuning-free importance evaluation using GradNorm, identifying and removing redundant layers in a targeted, segment-wise manner. Experimental results on both image classification tasks and large language models (LLMs) demonstrate that our proposed SGLP outperforms the state-of-the-art methods in accuracy and efficiency. Our approach achieves significant model compression with minimal performance degradation, making it well-suited for deployment in resource-limited environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。