arXiv:2608.06411cs.AIcs.CV2026-08

用动态选层预测注意力,高效剪裁视觉令牌。

Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Prunin

论文配图:Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Prunin
图 1 · 摘自论文原文
  • 根据问题动态选最优中间层,替代固定层做注意力预测。
  • 仅用5.56%视觉令牌,保持97.5%模型性能,提速3.09倍。
  • 轻量级预测器可嵌入现有加速流程,适合部署优化。

多模态大语言模型在多种视觉-语言任务中表现优异,但处理大量视觉令牌导致效率受限。视觉令牌剪枝可降低开销,但依赖准确的令牌重要性估计。近期研究发现,语言模型中间层的文本到视觉注意力能有效指导剪枝,通常采用预设中间层注意力来选择保留的视觉令牌。然而存在两个问题:第一,不同样本中对问题最敏感的层差异显著,固定层不优;第二,获取合适中间层注意力需处理多个语言模型层,计算成本已大量消耗。为此,本文提出中间层注意力预测(MAP),通过问题对比教师选择机制,对比原始问题与参考问题下的注意力,动态选定样本特定的教师层,并将该层注意力蒸馏为轻量预测器,从多模态输入特征中估计视觉令牌重要性。推理时,MAP结合预测重要性与多样性准则,在首个语言模型层前剪枝视觉令牌。因此,无需生成注意力图即可剪枝,且兼容现有推理加速技术。在LLaVA-NeXT-7B上的十项基准测试中,仅使用5.56%的视觉令牌即保留97.5%的原始模型性能,实现3.09倍端到端加速。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) achieve strong performance across diverse vision-language tasks, but their efficiency is limited by the cost of processing numerous visual tokens. Visual token pruning can reduce this cost, but requires accurate token importance estimates. Recent studies have demonstrated that text-to-vision attention from middle language model layers can effectively guide visual token pruning, typically using attention from a predefined middle layer to select the visual tokens to retain. Two problems therefore remain. First, our analysis shows that the layer whose attention is most responsive to the question varies substantially across samples, making a fixed layer suboptimal. Second, obtaining attention from the appropriate middle layer requires processing numerous visual tokens through several language model layers, by which point considerable computation has already been spent. To address both problems, we propose Middle-layer Attention Prediction (MAP), which uses Question Contrastive Teacher Selection to identify a sample-specific teacher layer by contrasting attention under the original and reference questions, and distills attention from the selected layer into a lightweight predictor that estimates visual token importance from multi-modal input features. During inference, MAP combines the predicted importance scores with a diversity criterion to prune visual tokens before the first language model layer. Thus, MAP requires no attention maps for pruning and remains compatible with existing inference acceleration techniques. Across ten benchmarks on LLaVA-NeXT-7B, MAP retains 97.5% of the unpruned model performance with only 5.56% of the visual tokens, yielding a 3.09x end-to-end speedup.

视觉剪枝注意力预测多模态推理加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。