通过视觉与文本语义协同剪枝，显著降低大模型推理开销。
ViTCoP: Accelerating Large Vision-Language Models via Visual and Textual Semantic Collaborative Pruning
- 在视觉编码器过滤冗余，在语言模型分步联合剪枝。
- 极端剪枝下仍保持领先性能，延迟与显存大幅下降。
- 适配FlashAttention，适合高效率部署的视觉语言任务。
大型视觉语言模型因视觉标记冗余导致计算成本高昂。现有视觉标记剪枝方法存在两类局限：在视觉编码器中过早丢失关键视觉信息，或在大语言模型中导致选中标记间信息冗余。为此，我们提出视觉与文本语义协同剪枝框架（ViTCoP），结合视觉编码器中的冗余过滤与基于大语言模型层级特性的分步联合剪枝，高效保留关键且多样化的视觉标记。同时，为兼容FlashAttention等加速技术，我们在大语言模型中引入K向量的L2范数作为标记重要性度量。在多个大型视觉语言模型上的实验表明，ViTCoP不仅在图像与视频理解任务上超越现有方法，达到当前最优性能，且显著降低推理延迟与GPU显存占用。尤其在极端剪枝率下，其性能优势更加明显。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) incur high computational costs due to significant redundancy in their visual tokens. To effectively reduce this cost, researchers have proposed various visual token pruning methods. However, existing methods are generally limited, either losing critical visual information prematurely due to pruning in the vision encoder, or leading to information redundancy among the selected tokens due to pruning in the Large Language Models (LLMs). To address these challenges, we propose a Visual and Textual Semantic Collaborative Pruning framework (ViTCoP) that combines redundancy filtering in the vision encoder with step-wise co-pruning within the LLM based on its hierarchical characteristics, to efficiently preserve critical and informationally diverse visual tokens. Meanwhile, to ensure compatibility with acceleration techniques like FlashAttention, we introduce the L2 norm of K-vectors as the token saliency metric in the LLM. Extensive experiments on various Large Vision-Language Models demonstrate that ViTCoP not only achieves state-of-the-art performance surpassing existing methods on both image and video understanding tasks, but also significantly reduces model inference latency and GPU memory consumption. Notably, its performance advantage over other methods becomes even more pronounced under extreme pruning rates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。