arXiv:2608.03580cs.CV2026-08

通过动态剪枝与视觉令牌自适应选择,高效压缩视觉语言模型。

SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models

论文配图:SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models
图 1 · 摘自论文原文
  • 基于文本-视觉注意力均值筛选重要视觉令牌,减少冗余干扰。
  • 按模块敏感度动态调整剪枝比例,保持整体性能稳定。
  • 在多模态基准上超越现有方法,适合资源受限设备部署。

尽管视觉语言模型(VLMs)在图文理解任务中表现优异,但其庞大的参数量带来显著计算开销,限制了在资源受限设备上的部署。虽然剪枝对大语言模型有效,但直接应用于VLMs会导致性能大幅下降,主要原因是冗余视觉令牌干扰重要性评估。为此,我们提出SlimVLM,一种结构化剪枝框架,在保持任务性能的同时压缩VLMs。引入自适应视觉令牌选择策略,利用平均文本-视觉注意力得分评估视觉令牌重要性,基于阈值移除冗余令牌,优化重要性计算。针对不同模块对稀疏度的容忍度差异,提出敏感度感知的动态剪枝机制,通过计算剪枝前后模块输出的线性重构误差,确定各模块合适剪枝比例,保障整体性能稳定。实验表明,SlimVLM在多个多模态基准上优于现有方法,达到当前最佳性能。

原文摘要 · Abstract (English)

While Vision-Language Models (VLMs) have demonstrated remarkable performance in processing and understanding both text and images, their large parameter sizes lead to significant computational overhead, limiting their deployment on resource-constrained devices. While pruning has been effective for compressing Large Language Models (LLMs), directly applying it to VLMs leads to significant performance drops, largely due to redundant visual tokens interfering with importance estimation. To this end, we propose SlimVLM, a structured pruning framework designed to compress VLMs while preserving their task performance. We introduce an adaptive visual token selection strategy for VLMs that leverages average text-to-visual attention scores to assess the importance of visual tokens, removing redundant ones during pruning based on a set threshold, thereby optimizing the importance calculation. Recognizing the varying tolerance to sparsity across different modules, we also propose a Sensitivity-aware dynamic pruning mechanism that determines the appropriate pruning ratio for each module by calculating the linear reconstruction error between the outputs of the pruned and unpruned modules, ensuring overall performance stability. Experimental results show that SlimVLM outperforms existing methods across multiple multimodal benchmarks, achieving state-of-the-art performance.

视觉语言模型模型剪枝高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。