通过直接筛选关键视觉标记，提升视觉语言模型推理效率与准确率。
Object-Centric Vision Token Pruning for Vision Language Models
- 基于对象中心思想，直接挑选最具代表性的视觉标记。
- 在任意剪枝比例下均保持最高推理准确率，且无需微调模型。
- 方法可解释性强，适用于主流视觉语言模型部署优化。
在视觉语言模型（VLMs）中，视觉标记数量多但信息分散，导致计算开销大。现有剪枝方法多依赖间接、无保障的方式，难以保证性能。本文提出OC-VTP，一种直接且有保障的视觉标记剪枝方法：仅需轻量级预训练小型对象中心剪枝器，即可插入现有VLM中，无需任何数据集上的微调。该方法通过最小化从选中标记重构原始未剪枝标记的误差，确保保留最具代表性的视觉标记。在任意视觉剪枝比例下，OC-VTP均能帮助主流VLM维持最高推理准确率，同时展现出良好可解释性。代码已开源。
原文摘要 · Abstract (English)
In Vision Language Models (VLMs), vision tokens are quantity-heavy yet information-dispersed compared with language tokens, thus consume too much unnecessary computation. Pruning redundant vision tokens for high VLM inference efficiency has been continuously studied but all existing methods resort to indirect and non-guaranteed ways. We propose OC-VTP, a direct and guaranteed approach to select the most representative vision tokens for high-efficiency yet accuracy-preserving VLM inference. Our OC-VTP requires merely light-weight pre-training of a small object-centric vision token pruner, which can then be inserted into existing VLMs, without fine-tuning of any models on any datasets. It is gauranteed that the most representative vision tokens are kept by minimizing the error in reconstructing the original unpruned tokens from the selected ones. Across any vision pruning ratios, i.e., inference efficiency, our OC-VTP consistently helps mainstream VLMs to preserve the highest inference accuracy. Our pruning also demonstrates interesting interpretability. Our codes are available at https://github.com/GarryLarry010131/OC-VTP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。