不训练即可提升视觉语言模型像素定位效率
CLIP Tricks You: Training-free Token Pruning for Efficient Pixel Grounding in Large VIsion-Language Models
- 基于文本引导的视觉标记剪枝，反向排序相似度保留关键区域
- 在多种标记预算下性能优于现有方法超5%，保持90%原精度
- 适合需要快速推理且无训练条件的视觉定位场景
大型视觉语言模型中，视觉标记通常占输入标记的大多数，导致显著计算开销。为解决此问题，近期研究探索剪枝冗余或信息量低的视觉标记以提升图像理解效率。然而，这些方法在像素定位任务中表现不佳，因标记重要性高度依赖输入文本。通过对CLIP的深入分析发现，指代表区域内的视觉标记往往与文本表示相似度较低。基于此洞察，我们提出LiteLVLM，一种无需训练、由文本引导的视觉标记剪枝策略，用于高效像素定位推理。通过反转CLIP的视觉-文本相似度排名，LiteLVLM有效保留覆盖指代表区域的视觉标记，并恢复上下文标记以实现清晰的前景-背景分离。大量实验表明，LiteLVLM在多种标记预算下性能显著优于现有方法，提升超过5%。无需任何训练或微调，其在22%加速下仍保持90%原始性能，并实现2.3倍内存减少。代码已开源于https://github.com/sejong-rcv/LiteLVLM。
原文摘要 · Abstract (English)
In large vision-language models, visual tokens typically constitute the majority of input tokens, leading to substantial computational overhead. To address this, recent studies have explored pruning redundant or less informative visual tokens for image understanding tasks. However, these methods struggle with pixel grounding tasks, where token importance is highly contingent on the input text. Through an in-depth analysis of CLIP, we observe that visual tokens within referent regions often exhibit low similarity to their textual representation. Motivated by this insight, we introduce LiteLVLM, a training-free, text-guided token pruning strategy for efficient pixel grounding inference. By reversing the ranking of CLIP's visual-text similarity, LiteLVLM effectively retains visual tokens covering the referent regions, while recovering context tokens to enable clear foreground-background separation. Extensive experiments demonstrate that LiteLVLM significantly outperforms existing methods by over 5% across diverse token budgets. Without any training or fine-tuning, LiteLVLM maintains 90% of the original performance with a 22% speedup and a 2.3X memory reduction. Our code is available at https://github.com/sejong-rcv/LiteLVLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。