arXiv:2606.24156cs.CV2026-06中稿 · ECCV被引 1

通过分离模型先验与任务注意力,提升多模态模型的视觉令牌压缩效率。

Accelerating Multimodal Large Language Models with Prior-Corrected Token Reduction

论文配图:Accelerating Multimodal Large Language Models with Prior-Corrected Token Reduction
图 1 · 摘自论文原文
  • 引入空令牌探测模型先验,单次前向传播分离任务相关与无关注意力。
  • 在极端令牌预算下,相比基线模型准确率提升2.3%以上,推理速度加快1.8倍。
  • 无需训练,适配多种多模态大模型和下游任务,部署友好。

视觉令牌压缩已成为加速多模态大语言模型(MLLMs)的有效策略。现有方法通常基于文本-视觉注意力得分对令牌进行排序剪枝。然而,我们发现注意力常被模型自身先验主导:即使无文本指令,MLLMs 也倾向于关注某些与任务无关的区域。这导致指令相关令牌的注意力得分被抑制,增加其在剪枝中被误删的风险。为此,我们提出无需训练的先验校正令牌压缩方法(PriorTR),显式分离任务相关注意力与模型先验。PriorTR估计先验注意力图,并与任务条件注意力分布对比,衡量每个视觉令牌带来的额外有用信息。关键创新在于,通过引入一个作为指令无关探针的空令牌,在单次前向传播中同时计算模型先验与任务后验注意力,避免重复传播。在多个多模态基准和不同MLLM上的大量实验表明,PriorTR在准确率与效率权衡上持续优于强基线,尤其在极端令牌预算下表现突出。

原文摘要 · Abstract (English)

Visual token reduction has emerged as an effective strategy for accelerating Multimodal Large Language Models (MLLMs). Many existing methods prune tokens by ranking text-visual attention scores. However, we show that attention is often dominated by a model-induced prior: even without textual instruction, MLLMs tend to focus on certain task-agnostic regions. Consequently, the attention scores of instruction-conditioned tokens are suppressed, increasing the risk that these tokens are discarded during pruning. To address this issue, we propose Prior-Corrected Token Reduction (PriorTR), a training-free token reduction method that explicitly separates task-conditioned attention from the model-induced prior. PriorTR estimates the attention map of the prior, and contrasts it with the task-conditioned attention distribution to measure the additional usable information contributed by each visual token. Importantly, PriorTR computes both the model-induced prior and the task-conditioned posterior within a single forward pass by introducing a null token that serves as an instruction-agnostic probe in the attention block. This design avoids duplicated propagation. Extensive experiments across multiple multimodal benchmarks and MLLMs demonstrate that PriorTR consistently improves the trade-off between accuracy and efficiency over strong training-free baselines, particularly under aggressive token budgets.

多模态令牌压缩注意力机制推理加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。