动态剪枝视觉令牌，92.6%压缩率下保持模型性能
A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models
- 基于人类认知设计动态剪枝框架，单次前向传播完成冗余令牌剔除
- 在自由问答任务上剪掉92.6%视觉令牌，平均性能与基线持平
- 显著降低计算开销，支持更高效微调，性能提升110%
视觉令牌压缩对大视觉语言模型（LVLM）高效处理高分辨率输入至关重要。现有方法多采用固定压缩比，难以适应不同复杂度场景，常导致重要视觉信息被误删，影响模型表现。为此，我们提出受人类认知启发的动态剪枝框架 GlimpsePrune，通过数据驱动的“瞥视”机制，在生成答案前单次前向传播中剔除无关视觉令牌。该方法可剪除92.6%的视觉令牌，同时在自由形式视觉问答任务上平均保持基线性能。更低的计算成本也使微调更高效：增强版 GlimpsePrune+ 在维持相近剪枝率的同时，性能达到基线的110%。本工作为构建更强大且高效的LVLM开辟了新路径。
原文摘要 · Abstract (English)
Visual token compression is critical for Large Vision-Language Models (LVLMs) to efficiently process high-resolution inputs. Existing methods that typically adopt fixed compression ratios cannot adapt to scenes of varying complexity, often causing imprecise pruning that discards informative visual tokens and results in degraded model performance. To address this issue, we introduce a dynamic pruning framework, GlimpsePrune, inspired by human cognition. It takes a data-driven ''glimpse'' and prunes irrelevant visual tokens in a single forward pass before answer generation. This approach prunes 92.6% of visual tokens while on average fully retaining the baseline performance on free-form VQA tasks. The reduced computational cost also enables more effective fine-tuning: an enhanced GlimpsePrune+ achieves 110% of the baseline performance while maintaining a similarly high pruning rate. Our work paves a new way for building more powerful and efficient LVLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。