通过分层分头剪枝，让视觉语言模型更快更省显存。
PLPHP: Per-Layer Per-Head Vision Token Pruning for Efficient Large Vision-Language Models
- 按层按注意力头动态剪枝，保留关键视觉信息
- 推理速度提升18%，KV缓存减少50%以上
- 适合部署大模型的高效推理场景
大型视觉语言模型（LVLMs）在多模态任务中表现出色，但其推理效率受限于解码过程中处理的大量视觉标记。为此，我们提出分层分头视觉标记剪枝（PLPHP），一种两级细粒度剪枝方法，包括层级保留率分配和头级别视觉标记剪枝。受解码器各层间视觉标记重关注现象启发，动态调整各层的标记保留率：对视觉注意力强的层保留更多标记，低注意力层则大幅剪枝。此外，PLPHP在注意力头层级实施剪枝，使同一层内不同头可独立保留关键上下文。在多个基准上的实验表明，PLPHP实现18%的解码速度提升，键值缓存（KV Cache）大小减少超过50%，仅带来0.46%的平均性能下降，并在多图像任务中取得显著性能提升。结果验证了细粒度剪枝的有效性，推动了LVLM的高效性与可扩展性。代码将公开。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across a range of multimodal tasks. However, their inference efficiency is constrained by the large number of visual tokens processed during decoding. To address this challenge, we propose Per-Layer Per-Head Vision Token Pruning (PLPHP), a two-level fine-grained pruning method including Layer-Level Retention Rate Allocation and Head-Level Vision Token Pruning. Motivated by the Vision Token Re-attention phenomenon across decoder layers, we dynamically adjust token retention rates layer by layer. Layers that exhibit stronger attention to visual information preserve more vision tokens, while layers with lower vision attention are aggressively pruned. Furthermore, PLPHP applies pruning at the attention head level, enabling different heads within the same layer to independently retain critical context. Experiments on multiple benchmarks demonstrate that PLPHP delivers an 18% faster decoding speed and reduces the Key-Value Cache (KV Cache) size by over 50%, all at the cost of 0.46% average performance drop, while also achieving notable performance improvements in multi-image tasks. These results highlight the effectiveness of fine-grained token pruning and contribute to advancing the efficiency and scalability of LVLMs. Our source code will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。