通过自适应分两阶段剪枝，高效减少视觉冗余令牌。
ERASE: Eliminating Redundant Visual Tokens via Adaptive Two-Stage Token Pruning
- 根据图像复杂度动态调整剪枝策略，识别关键视觉特征。
- 在85%剪枝率下保持模型90%以上准确率，优于现有方法。
- 适合需要高效多模态推理的场景，如高分辨率图像理解。
近期视觉语言模型（VLMs）的发展使大语言模型（LLMs）能够处理高分辨率图像，显著提升现实世界中的多模态理解能力。然而，这一能力引入大量视觉令牌，带来巨大计算开销。为缓解此问题，已有多种视觉令牌剪枝方法被提出。但现有方法主要依赖模型内部学习到的语义特征来捕捉视觉冗余，且缺乏根据输入图像复杂度自适应调整剪枝策略的能力。本文提出ERASE，一种两阶段视觉令牌剪枝框架，通过适应图像复杂度的剪枝策略保留显著令牌。实验表明，ERASE在大幅减少视觉令牌的同时维持高精度。对于Qwen2.5-VL-7B模型，在85%的令牌剪枝率下，保留了89.46%的原始模型准确率，而最优先前方法仅达78.1%。代码已公开于https://github.com/Tuna-Luna/ERASE。
原文摘要 · Abstract (English)
Recent advancements in Vision-Language Models (VLMs) enable large language models (LLMs) to process high-resolution images, significantly improving real-world multimodal understanding. However, this capability introduces a large number of vision tokens, resulting in substantial computational overhead. To mitigate this issue, various vision token pruning methods have been proposed. Nevertheless, existing approaches predominantly rely on learned semantic features within the model to capture visual redundancy. Moreover, they lack adaptive mechanisms to adjust pruning strategies according to the complexity of the input image. In this paper, we propose ERASE, a two-stage vision token pruning framework that identifies and retains salient tokens through pruning strategies adaptive to image complexity. Experiment results demonstrate that ERASE significantly reduces vision tokens while preserving accuracy. For Qwen2.5-VL-7B, at a token pruning ratio of 85\%, ERASE retains 89.46% of the original model accuracy, whereas the best prior method retains only 78.1%. Our code is available at https://github.com/Tuna-Luna/ERASE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。