通过剪枝冗余层，实现大模型压缩加速而不损失性能。
Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layers
- 基于视觉语言令牌重要性与层间特征差距设计新剪枝框架。
- 在不微调情况下实现高效率压缩，保持原模型精度。
- 适合需要快速部署大模型的开发者与实际应用者。
尽管大型视觉语言模型（LVLMs）在多模态理解与推理方面展现出强大能力，但其实际应用仍受限于庞大的参数量和高昂的计算成本。自然语言处理领域近期提出的层剪枝方法为无训练压缩提供了可能，但由于视觉与语言模态差异，这些方法在LVLM中效果不佳。本文通过大量实验发现，非关键视觉-语言（VL）标记和层间特征差异是剪枝的主要障碍。基于此，我们提出Short-LVLM（SVL）框架，利用重要VL标记并缓解层间特征差距。该方法无需训练、兼容多种模型，实现了性能与效率的优异平衡。代码已开源。
原文摘要 · Abstract (English)
Although large vision-language models (LVLMs) have demonstrated impressive capabilities in multi-modal understanding and reasoning, their practical applications are still limited by massive model parameters and high computational costs. Recent efforts from natural language processing (NLP) have shown the effectiveness of layer pruning, offering a plausible training-free compression solution. However, due to the modality divergence between vision and language, it is unclear whether these NLP techniques are still effective in LVLMs. In this paper, we empirically prove that directly applying these layer pruning methods to LVLMs is ineffective. Through extensive experiments, we find that non-essential vision-language (VL) tokens and inter-layer feature gaps pose critical challenges to pruning layers in LVLMs. Based on these insights, we propose a novel framework Short-LVLM (SVL) that can utilize important VL tokens and mitigate the layer-wise feature gaps. Notably, Short-LVLM not only achieves a superior trade-off between performance and efficiency but also exhibits several potential advantages, i.e., training-free, model-agnostic, and highly compatible. The code for this work is publicly available at https://github.com/ASGO-MM/Short-LVLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。