通过筛选关键视觉特征,让视觉语言模型更高效且不损失性能。
VisionZip: Longer is Better but Not Necessary in Vision Language Models
- 仅保留重要视觉标记,减少冗余以提升效率。
- 在多数任务上性能比之前最优方法高至少5%。
- 适合真实场景多轮对话,推理速度提升8倍。
近期视觉语言模型通过增加视觉标记长度提升了性能,但导致计算成本显著上升。我们发现主流视觉编码器(如CLIP和SigLIP)生成的视觉标记存在显著冗余。为此,我们提出VisionZip,一种简单有效的标记选择方法,仅输入对语言模型有贡献的代表性视觉标记,降低冗余、提升效率并保持性能。该方法适用于图像与视频理解任务,尤其擅长真实场景下的多轮对话。实验表明,VisionZip在几乎所有设置中均优于此前最先进方法,性能提升至少5%;同时显著加快推理速度,使LLaVA-Next 13B的预填充时间缩短8倍,并实现比LLaVA-Next 7B更快的推理速度与更优结果。我们还分析了冗余成因,呼吁社区关注更优视觉特征提取而非单纯延长标记长度。代码已开源:https://github.com/dvlab-research/VisionZip。
原文摘要 · Abstract (English)
Recent advancements in vision-language models have enhanced performance by increasing the length of visual tokens, making them much longer than text tokens and significantly raising computational costs. However, we observe that the visual tokens generated by popular vision encoders, such as CLIP and SigLIP, contain significant redundancy. To address this, we introduce VisionZip, a simple yet effective method that selects a set of informative tokens for input to the language model, reducing visual token redundancy and improving efficiency while maintaining model performance. The proposed VisionZip can be widely applied to image and video understanding tasks and is well-suited for multi-turn dialogues in real-world scenarios, where previous methods tend to underperform. Experimental results show that VisionZip outperforms the previous state-of-the-art method by at least 5% performance gains across nearly all settings. Moreover, our method significantly enhances model inference speed, improving the prefilling time by 8x and enabling the LLaVA-Next 13B model to infer faster than the LLaVA-Next 7B model while achieving better results. Furthermore, we analyze the causes of this redundancy and encourage the community to focus on extracting better visual features rather than merely increasing token length. Our code is available at https://github.com/dvlab-research/VisionZip .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。