通过文本引导筛选视觉标记，实现高效无训练的视觉语言模型推理优化。
SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
- 用文本注意力矩阵评估视觉标记重要性，动态剪枝冗余视觉标记。
- 使LLaVA在保持97%准确率前提下，降低54%计算量与37%延迟。
- 无需额外参数或微调，适合部署资源受限场景的视觉语言模型。
在视觉语言模型中，视觉标记虽信息稀疏却带来大量计算开销。现有方法通常需训练网络来剪枝冗余视觉标记，而本文提出无需训练的SparseVLM机制，利用文本标记与视觉标记间的自注意力关系，判断视觉标记重要性并进行剪枝，以最大化稀疏性同时保留关键信息。我们设计基于排名的分层稀疏比例自适应策略，并引入标记回收方法将被剪枝标记压缩为更紧凑表示。实验表明，该方法显著提升多种视觉语言模型效率。例如，集成SparseVLM的LLaVA在图像和视频理解任务中实现54%的浮点运算减少（FLOPs）、37%的CUDA延迟下降，同时维持97%原始准确率。代码已开源。
原文摘要 · Abstract (English)
In vision-language models (VLMs), visual tokens usually bear a significant amount of computational overhead despite sparsity of information in them when compared to text tokens. To address this, most existing methods learn a network to prune redundant visual tokens using certain training data. Differently, we propose a text-guided training-free token optimization mechanism dubbed SparseVLM that eliminates the need of extra parameters or fine-tuning costs. Given that visual tokens complement text tokens in VLM's linguistic reasoning, we select relevant text tokens to rate the significance of visual tokens using self-attention matrices and, then, prune visual tokens using the proposed strategy to maximize sparsity while retaining information. In particular, we introduce a rank-based strategy to adaptively determine the sparsification ratio for each layer, alongside a token recycling method that compresses pruned tokens into more compact representations. Experimental results show that SparseVLM increases the efficiency of various VLMs in a number of image and video understanding tasks. For example, LLaVA when equipped with SparseVLM achieves 54% reduction in FLOPs, 37% decrease in CUDA latency while maintaining 97% of its original accuracy. Our code is available at https://github.com/Gumpest/SparseVLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。