通过交错剪枝与微调，让大模型在删掉四分之一层后仍保持高精度。
INTERLACE: Interleaved Layer Pruning and Efficient Adaptation in Large Vision-Language Models
- 按三连层分析冗余，剪掉前两层中较冗余的一层，保留一层微调补能力。
- 仅用1%数据微调1轮，剪枝25%后仍达88.9%性能保留率，领先现有方法。
- 适合资源受限场景下的大视觉语言模型高效部署与适配。
我们提出INTERLACE，一种新型框架，在剪枝大视觉语言模型（VLMs）冗余层的同时，通过样本高效的微调维持性能。现有层剪枝方法在应用于VLM时会导致显著性能下降。为此，我们分析连续三层层的局部冗余性，剪掉前两层中冗余度更高的那一层，对剩余层进行微调以补偿损失容量，并冻结第三层作为微调过程中的稳定锚点。该交错微调-冻结设计使模型能在剪枝后快速收敛，且所需数据极少。仅在FineVision数据集1%的数据上微调一个周期，即可实现剪枝25%后88.9%的平均性能保留，达到当前最优水平。代码已开源：https://github.com/pmadinei/Interlace.git。
原文摘要 · Abstract (English)
We introduce INTERLACE, a novel framework that prunes redundant layers in VLMs while maintaining performance through sample-efficient finetuning. Existing layer pruning methods lead to significant performance drop when applied to VLMs. Instead, we analyze triplets of consecutive layers to identify local redundancy, removing the most redundant of the first two layers, finetune the remaining layer to compensate for the lost capacity, and freeze the third layer to serve as a stable anchor during finetuning. We found that this interleaved finetune-freeze design enables rapid convergence with minimal data after pruning. By finetuning only a subset of layers on just 1% of the FineVision dataset for one epoch, Interlace achieves 88.9% average performance retention after dropping 25% of the network, achieving SOTA performance. Our code is available at: https://github.com/pmadinei/Interlace.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。