按操作符粒度跳过冗余视觉计算，提升多模态大模型推理效率。
Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference
- 将Transformer层拆解为注意力与前馈操作，识别冗余计算点
- 在保持完整视觉序列下，减少33.7%计算量，精度损失仅0.5%
- 适用于需高效推理的多模态模型，尤其适合长视觉输入场景
多模态大语言模型（MLLMs）处理长视觉标记序列时，推理计算量显著增加。现有加速方法通常整体删除视觉标记或跳过整层更新，但这类粗粒度策略可能误删关键信息或抑制有效操作。本文从答案可观察角度研究视觉标记计算，发现后期视觉标记更新对答案表示影响甚微。基于此，我们分解每个Transformer层为注意力与前馈网络（FFN）操作，发现有效视觉计算常呈现操作主导且层依赖特性。提出一种操作符级视觉标记跳过框架，在保留完整视觉序列的同时，选择性跳过冗余的注意力、FFN或两者。在三种MLLM架构和10个VQA基准上实验表明，该方法实现良好效率-精度权衡：在Qwen3-VL上降低33.7% TFLOPs，性能仍达原模型的99.5%。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) increasingly process long visual-token sequences, increasing the overall inference computation. Existing acceleration methods usually remove visual tokens or skip visual-token updates in entire layers, but these coarse strategies may discard fine-grained evidence or suppress useful operators together with redundant ones. In this paper, we study visual-token computation from an answer-observable perspective and find that late visual-token updates can remain large while having little effect on answer-token representations. Motivated by this answer-silent redundancy, we decompose each Transformer layer into attention and FFN operators and show that useful visual computation is often operator-dominant and layer-dependent. We propose an operator-level visual-token skipping framework that preserves the full visual-token sequence while selectively bypassing redundant attention, FFN, or both. Experiments across three MLLM architectures and 10 VQA benchmarks show that our method achieves strong efficiency-accuracy trade-offs, reducing \textbf{33.7\%} TFLOPs on Qwen3-VL while retaining \textbf{99.5\%} of the vanilla model performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。