只保留问题相关的图像区域，减少冗余计算
CROP: Contextual Region-Oriented Visual Token Pruning
- 先定位问题相关图像区域，再按需压缩或剪枝
- 在多个VQA任务上超越现有剪枝方法，性能领先
- 无需训练，适合部署在资源受限的视觉语言模型中
当前基于视觉语言模型（VLM）的视觉问答方法通常处理整张图像，导致大量冗余视觉标记，显著增加内存与计算开销。为此，本文提出上下文区域导向的视觉标记剪枝框架CROP，通过定位与剪枝两步实现压缩：首先利用高效模型识别与输入问题相关的上下文区域；随后引入两种剪枝策略：(1) 预大模型压缩（PLC），自适应地以不同比例压缩不同图像区域；(2) 内大模型剪枝（ILP），一种无需训练的方法，在大模型早期层中依据识别出的上下文区域剪枝视觉标记。在多种VQA任务上的实验表明，CROP显著优于现有视觉标记剪枝方法，达到当前最优性能。
原文摘要 · Abstract (English)
Current VLM-based VQA methods often process entire images, leading to excessive visual tokens that include redundant information irrelevant to the posed question. This abundance of unnecessary image details creates numerous visual tokens, drastically increasing memory and computational requirements in VLMs. To address this, we propose Contextual Region-Oriented Visual Token Pruning (CROP), a novel framework to compress visual tokens through a two-step process: Localization and Pruning. Specifically, CROP first employs an efficient model to identify the contextual region relevant to the input query. Subsequently, two distinct strategies are introduced for pruning: (1) Pre-LLM Compression (PLC), which adaptively compresses different image regions with varying ratios, and (2) Inner-LLM Pruning (ILP), a training-free method that prunes tokens within early LLM layers guided by the identified contextual region. Extensive experiments on a wide range of VQA tasks demonstrate that CROP significantly outperforms existing visual token pruning methods and achieves state-of-the-art performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。