通过信息流分析优化视觉语言模型的冗余令牌剪枝，提升效率与性能。
FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models
- 基于信息流建模跨层交互，识别动态冗余视觉令牌
- 在LLaVA-1.5上减少88.9%令牌，性能领先1.6%
- 适用于追求高效多模态推理的研究者与开发者
大型视觉语言模型（LVLM）在多模态理解方面表现优异，但因视觉令牌冗余导致计算开销高。现有剪枝方法通常依赖单层注意力得分来排序并剪除冗余视觉令牌。然而，由于令牌与层间交互复杂，这引发一个根本问题：单一层次的准则是否足以识别冗余？为此，我们从信息流的角度重新思考冗余的产生机制，信息流通过捕捉跨层的令牌间信息传递来建模其交互。我们发现：(1) CLS令牌充当信息中继，可简化复杂流分析；(2) 冗余随层逐步动态出现，表现为注意力集中；(3) 单层注意力得分可能导致冗余判定矛盾。基于此，提出FlowCut——一种面向信息流的剪枝框架，弥补当前准则的不足，更契合模型内在行为。大量实验表明，FlowCut在LLaVA-1.5-7B上实现88.9%令牌削减，性能领先最先进方法1.6%；在LLaVA-NeXT-7B上实现94.4%缩减，领先4.3%，预填充阶段提速3.2倍。代码已开源。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) excel at multimodal understanding but suffer from high computational costs due to redundant vision tokens. Existing pruning methods typically rely on single-layer attention scores to rank and prune redundant visual tokens to solve this inefficiency. However, as the interaction between tokens and layers is complicated, this raises a basic question: Is such a simple single-layer criterion sufficient to identify redundancy? To answer this question, we rethink the emergence of redundant visual tokens from a fundamental perspective: information flow, which models the interaction between tokens and layers by capturing how information moves between tokens across layers. We find (1) the CLS token acts as an information relay, which can simplify the complicated flow analysis; (2) the redundancy emerges progressively and dynamically via layer-wise attention concentration; and (3) relying solely on attention scores from single layers can lead to contradictory redundancy identification. Based on this, we propose FlowCut, an information-flow-aware pruning framework, mitigating the insufficiency of the current criterion for identifying redundant tokens and better aligning with the model's inherent behaviors. Extensive experiments show that FlowCut achieves superior results, outperforming SoTA by 1.6% on LLaVA-1.5-7B with 88.9% token reduction, and by 4.3% on LLaVA-NeXT-7B with 94.4% reduction, delivering 3.2x speed-up in the prefilling stage. Our code is available at https://github.com/TungChintao/FlowCut
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。