arXiv:2503.20540cs.CV2025-03被引 8

通过语言反馈分析视觉冗余,实现高效图像令牌压缩。

Beyond Intermediate States: Explaining Visual Redundancy through Language

  • 从视觉-语言关联与注意力分布双视角分析冗余
  • 仅保留10%-20%令牌仍达90%-110%原性能
  • 适合需要降低视觉计算负担的多模态应用

多模态大模型常处理数千个视觉令牌,消耗大量上下文窗口并带来显著计算开销。以往基于中间状态(如注意力分数)的令牌剪枝方法因无法捕捉视觉令牌对模型视觉理解的影响(即文本候选概率),难以精准定义视觉冗余。本文通过操控视觉输入,从令牌中心和上下文中心两个角度分析文本输出变化,实现直观全面的分析。实验发现,低ViT-[cls]关联度和低图文注意力得分的视觉令牌仍包含可识别信息,并对整体图像信息有显著贡献。为此,我们融合双重视角,引入上下文无关条件,从训练图像中识别冗余原型,从而在推理时检测每个视觉令牌的冗余性。在单图、多图及视频理解任务上的大量实验表明,该方法有效,在剪除80%-90%视觉令牌的同时,性能保持在90%-110%之间。

原文摘要 · Abstract (English)

Multi-modal Large Langue Models (MLLMs) often process thousands of visual tokens, which consume a significant portion of the context window and impose a substantial computational burden. Prior work has empirically explored visual token pruning methods based on MLLMs' intermediate states (e.g., attention scores). However, they have limitations in precisely defining visual redundancy due to their inability to capture the influence of visual tokens on MLLMs' visual understanding (i.e., the predicted probabilities for textual token candidates). To address this issue, we manipulate the visual input and investigate variations in the textual output from both token-centric and context-centric perspectives, achieving intuitive and comprehensive analysis. Experimental results reveal that visual tokens with low ViT-[cls] association and low text-to-image attention scores can contain recognizable information and significantly contribute to images' overall information. To develop a more reliable method for identifying and pruning redundant visual tokens, we integrate these two perspectives and introduce a context-independent condition to identify redundant prototypes from training images, which probes the redundancy of each visual token during inference. Extensive experiments on single-image, multi-image and video comprehension tasks demonstrate the effectiveness of our method, notably achieving 90% to 110% of the performance while pruning 80% to 90% of visual tokens.

多模态视觉压缩令牌剪枝语言反馈

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。