提出无需训练的视觉标记剪枝方法,显著降低大模型推理开销。
E2S-Pruner: Progressive Two-Stage Evidence Fusion for Visual Token Pruning in Vision-Language Models

- 分两阶段融合注意力头与层间证据,评估标记重要性
- 保留98%性能仅需192个视觉标记,吞吐提升2倍以上
- 适用于多模型且可避免标记集中于局部区域
视觉语言模型通常将图像编码为数百个视觉标记,导致显著的推理延迟和显存开销。现有剪枝方法主要依赖注意力分数,并直接聚合各注意力头与网络层输出,难以刻画证据不确定性与冲突。我们提出E2S-Pruner,一种无需额外模型、可训练参数或微调的渐进式两阶段证据融合视觉标记剪枝框架。第一阶段将每个注意力头视为独立证据源,基于证据清晰度与头间一致性估计其可靠性,用重要、不重要、不确定三状态表示每个视觉标记。第二阶段采用Dempster–Shafer证据理论量化层间冲突,融合多层互补证据。进一步引入空间新颖性约束,促进不同图像区域的覆盖,防止保留标记集中在少数显著区域。在LLaVA-1.5-7B上,当平均保留视觉标记数分别为192、128、64时,分别保持98.0%、96.8%、90.6%的综合性能,同时在128标记和64标记设置下吞吐分别提升1.96倍和2.09倍。Qwen2-VL-7B实验进一步验证了跨模型泛化能力。代码已开源。
原文摘要 · Abstract (English)
Vision-language models typically encode an image into hundreds of visual tokens, incurring substantial inference latency and GPU memory overhead. Existing pruning methods largely rely on attention scores and directly aggregate outputs across attention heads and network layers, making it difficult to characterize evidential uncertainty and conflict. We propose E2S-Pruner, a progressive two-stage evidence-fusion framework for visual token pruning that requires no auxiliary model, trainable parameters, or fine-tuning. In the first stage, E2S-Pruner treats each attention head as an independent evidence source, estimates its reliability from evidence clarity and inter-head consistency, and represents each visual token using three states: important, unimportant, and uncertain. In the second stage, Dempster--Shafer evidence theory is used to quantify inter-layer conflict and fuse complementary evidence from multiple network layers. We further introduce a spatial novelty constraint that promotes coverage of distinct image regions and prevents the retained tokens from concentrating in a few locally salient areas. On LLaVA-1.5-7B, E2S-Pruner retains 98.0%, 96.8%, and 90.6% of the aggregate performance when the average numbers of retained visual tokens are 192, 128, and 64, respectively, while improving throughput by 1.96x and 2.09x under the 128-token and 64-token settings. Experiments on Qwen2-VL-7B further demonstrate cross-model generalization. Code is available at https://github.com/taoyu-qian/E2S-Pruner.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。