通过跳过冗余视觉注意力,让多模态大模型更高效推理
Look Less, Reason More: Block-wise Attention Skipping for Efficient Multimodal LLMs

- 按块跳过深层视觉自注意力,保留关键语义演进
- 在多种模型上实现94.16%~100.31%性能保留
- 无需训练,适配不同任务的动态推理深度
多模态大语言模型因长视觉序列的自注意力计算呈二次增长而面临严重推理瓶颈。我们发现当前架构存在关键低效:视觉注意力饱和。分析表明,视觉标记在早期层即快速建立空间结构和模态内关系,导致深层视觉-视觉自注意力计算冗余。相反,深层前馈网络对将视觉特征投影到动态文本语义空间仍至关重要。基于此,我们提出无需训练的推理范式Visual-Skip(V-Skip),将空间交互与语义演进解耦。不丢弃标记,而是通过选择性跳过饱和的视觉自注意力模块,实现块级结构化稀疏。此外,考虑到下游任务需不同推理深度,V-Skip采用轻量少样本校准,动态路由最优稀疏路径。大量实验表明,V-Skip能有效跳过冗余视觉注意力,实现块级稀疏,在多种多模态大模型上保持94.16%至100.31%性能。最终证明,要更有效地推理,模型无需舍弃所见——只需在合适深度‘看得更少’。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) face a significant inference bottleneck due to the quadratic computational cost of self-attention over long visual token sequences. However, we identify a critical inefficiency in current architectures: Visual Attention Saturation. Our analysis reveals that visual tokens rapidly establish their spatial structure and intra-modal relationships in early layers, rendering visual-to-visual self-attention in deeper layers computationally redundant. Conversely, Feed-Forward Networks (FFNs) in these layers remain essential for projecting visual features into the evolving textual semantic space. Leveraging this insight, we present Visual-Skip (V-Skip), a training-free inference paradigm that decouples spatial interaction from semantic evolution. Rather than discarding tokens, V-Skip imposes block-wise structured sparsity by selectively bypassing saturated visual self-attention modules. Furthermore, recognizing that varying downstream tasks demand distinct reasoning depths, V-Skip employs a lightweight, few-shot calibration to dynamically route the task-optimal sparsity path. Extensive experiments demonstrate that V-Skip effectively bypasses redundant vision attention to achieve block-wise sparsity, maintaining a 94.16% to 100.31% performance retention across diverse MLLMs. Ultimately, we prove that to reason more effectively, models do not need to discard what they see -- they simply need to "look less" at the right depth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。