arXiv:2505.12359cs.LGcs.CV2025-05被引 2

通过分阶段注意力引导,高效减少视觉语言模型推理冗余

STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inference

  • 分两阶段:先用自注意力删低层冗余特征,再用跨模态注意力去无关信息
  • 在高剪枝率下仍保持性能,部分任务反而提升
  • 无需训练、即插即用,适用于多种视觉语言模型

尽管大型视觉语言模型(LVLM)利用丰富的视觉标记表示在多模态任务中表现优异,但这些标记在推理过程中也带来了显著的计算开销。现有的无训练标记剪枝方法通常采用单阶段策略,仅关注视觉自注意力或视觉-文本交叉注意力,这种局部视角常忽略模型内部的信息流动,导致在高剪枝率下性能大幅下降。本文提出STAR(分阶段注意力引导的标记剪枝),一种无训练、即插即用的全局剪枝框架。STAR在两个互补阶段进行注意力引导的剪枝:早期基于视觉自注意力剔除冗余的低层特征,后期基于跨模态注意力移除任务无关标记。该整体方法显著降低计算成本,同时更好地保留关键任务信息。在多个LVLM架构和基准上的大量实验表明,STAR在实现显著加速的同时,保持相当甚至部分提升的性能。

原文摘要 · Abstract (English)

Although large vision-language models (LVLMs) leverage rich visual token representations to achieve strong performance on multimodal tasks, these tokens also introduce significant computational overhead during inference. Existing training-free token pruning methods typically adopt a single-stage strategy, focusing either on visual self-attention or visual-textual cross-attention. However, such localized perspectives often overlook the broader information flow across the model, leading to substantial performance degradation, especially under high pruning ratios. In this work, we propose STAR (Stage-wise Attention-guided token Reduction), a training-free, plug-and-play framework that approaches token pruning from a global perspective. Instead of pruning at a single point, STAR performs attention-guided reduction in two complementary stages: an early-stage pruning based on visual self-attention to remove redundant low-level features, and a later-stage pruning guided by cross-modal attention to discard task-irrelevant tokens. This holistic approach allows STAR to significantly reduce computational cost while better preserving task-critical information. Extensive experiments across multiple LVLM architectures and benchmarks show that STAR achieves strong acceleration while maintaining comparable, and in some cases even improved performance.

视觉语言模型推理加速标记剪枝注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。