通过视觉信息流优化,实现90%视觉令牌压缩而性能几乎不变。
VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimization
- 基于注意力上下文与局部熵计算令牌重要性
- 支持逐级修剪并回收被删令牌避免信息损失
- 适配不同模型的动态优化策略,适合部署加速
大型多模态模型(LMMs)依赖大量视觉令牌以实现精细视觉理解,但导致显著计算开销。现有方法通常基于纯视觉或跨模态注意力分数生成重要性图,进行单阶段或多阶段修剪,但策略简单,常引发性能大幅下降。本文提出VFlowOpt,一种引入重要性图生成与渐进式修剪模块的剪枝框架,并加入回收机制。通过计算图像令牌的注意力上下文相关性与块级信息熵构建重要性图,决定保留或剪除令牌,并将剪除的令牌聚合为可复用的回收令牌,防止信息丢失。进一步提出基于视觉信息流的优化方法,将LMM中最后一个令牌视为文本-视觉交互最具代表性的信号,最小化剪枝前后令牌表示差异,从而为不同模型定制更优剪枝策略。实验表明,VFlowOpt可在保留相近性能的前提下剪除90%视觉令牌,实现KV-Cache内存减少89%,推理速度提升3.8倍。
原文摘要 · Abstract (English)
Large Multimodal Models (LMMs) excel in visual-language tasks by leveraging numerous visual tokens for fine-grained visual information, but this token redundancy results in significant computational costs. Previous research aimed at reducing visual tokens during inference typically leverages importance maps derived from attention scores among vision-only tokens or vision-language tokens to prune tokens across one or multiple pruning stages. Despite this progress, pruning frameworks and strategies remain simplistic and insufficiently explored, often resulting in substantial performance degradation. In this paper, we propose VFlowOpt, a token pruning framework that introduces an importance map derivation process and a progressive pruning module with a recycling mechanism. The hyperparameters of its pruning strategy are further optimized by a visual information flow-guided method. Specifically, we compute an importance map for image tokens based on their attention-derived context relevance and patch-level information entropy. We then decide which tokens to retain or prune and aggregate the pruned ones as recycled tokens to avoid potential information loss. Finally, we apply a visual information flow-guided method that regards the last token in the LMM as the most representative signal of text-visual interactions. This method minimizes the discrepancy between token representations in LMMs with and without pruning, thereby enabling superior pruning strategies tailored to different LMMs. Experiments demonstrate that VFlowOpt can prune 90% of visual tokens while maintaining comparable performance, leading to an 89% reduction in KV-Cache memory and 3.8 times faster inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。