提出单次前向视觉令牌剪枝方法,显著提升高分辨率多模态模型推理速度。
Structured Redundancy Modeling for Efficient Visual Token Pruning in High-Resolution MLLMs

- 通过结构化冗余建模,实现单次前向计算完成重要性筛选
- 在512个令牌下将选择耗时从112.4毫秒降至2.5毫秒,减少110毫秒
- 适合需要低延迟推理的高分辨率多模态应用,如视觉理解与生成
近期高分辨率多模态大模型(MLLMs)每输入生成数千个视觉令牌,导致视觉令牌爆炸,引发严重延迟瓶颈。尽管令牌剪枝可缓解此问题,但现有基于子集优化的方法通常依赖迭代构造以同时捕捉视觉多样性与指令相关性。随着令牌数量增长,这种序列依赖带来显著选择开销,严重限制理论计算量减少向实际运行时间加速的转化。为此,我们提出单次前向剪枝器(SFPruner),将冗余控制直接嵌入评分空间,避免迭代组合优化。该非迭代框架通过两个互补机制,在单次前向传播中实现冗余感知的重要性选择:首先,引入语义引导的岭杠杆方案,融合指令相关性与视觉显著性,抑制主导协方差方向,缓解表示偏差;其次,基于排名的方向掩码通过不对称相似性竞争,使高分令牌通过并行张量操作显式抑制冗余低分候选。大量实验表明,本方法保持稳定的选择成本,将令牌选择过程缩短最多110毫秒(从112.4毫秒降至2.5毫秒,512令牌下),成功将理论令牌压缩转化为实际推理加速,同时在极端压缩下仍保持与顶尖技术相当的性能。
原文摘要 · Abstract (English)
Recent high-resolution Multimodal Large Language Models (MLLMs) generate thousands of visual tokens per input, leading to a visual token explosion that introduces severe latency bottlenecks. While token pruning mitigates this issue, state-of-the-art subset-optimization methods typically rely on iterative subset construction to jointly capture visual diversity and instruction relevance. As visual token counts scale, this sequential dependency introduces significant selection overhead, severely limiting the translation of theoretical FLOPs reductions into actual wall-clock speedups. To address this limitation, we propose Single-Forward Pruner (SFPruner), a structural reformulation of visual token pruning that embeds redundancy control directly into the scoring space, bypassing the need for iterative combinatorial optimization. Our non-iterative framework achieves redundancy-aware importance selection in a single forward pass through two complementary mechanisms. First, to attenuate redundancy at the covariance level, we introduce a semantics-guided ridge leverage scheme. By integrating instruction relevance and visual saliency, this mechanism suppresses dominant covariance directions and mitigates representation bias. Second, ranking-based directional masking resolves residual overlap through asymmetric similarity competition, where higher-scoring tokens explicitly suppress redundant lower-scoring alternatives via parallel tensor operations. Extensive evaluations demonstrate that our approach maintains stable selection costs, reducing the token selection process by up to 110 ms, from 112.4 ms to just 2.5 ms at 512 tokens in Qwen2.5-VL. This structural efficiency successfully translates theoretical token reductions into tangible inference speedups while preserving highly competitive performance against state-of-the-art techniques under aggressive compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。