提出无需训练的剪枝方法，显著降低视觉语言模型计算量
ASAP: Attention-Shift-Aware Pruning for Efficient LVLM Inference
- 用动态双向软注意力掩码缓解注意力偏移问题
- 通过语义相似性合并冗余视觉标记，保留99.02%性能
- 兼容KV缓存，适合部署在资源受限设备
大型视觉语言模型（LVLMs）虽具备卓越多模态能力，但处理高分辨率视觉标记时存在二次方计算开销，构成关键瓶颈。现有令牌压缩方法未能充分挖掘注意力值，且忽视了LVLM中固有的“注意力偏移”现象，导致选择不具信息量的标记。本文提出ASAP，一种无需训练、兼容KV缓存的剪枝方案。首先，通过动态双向软注意力掩码缓解注意力偏移，确保选择真正有意义的标记；其次，鉴于标记集合内存在高语义冗余，引入加权软合并组件，合并语义相近的标记，仅保留最具特征密度的视觉区块供后续层使用。ASAP实现近乎无损的视觉上下文压缩，在保持LLaVA-NeXT-7B原模型99.02%性能的同时，将计算量（FLOPs）降低约80%。
原文摘要 · Abstract (English)
While Large Vision-Language Models (LVLMs) demonstrate exceptional multi-modal capabilities, the quadratic computational cost of processing high-resolution visual tokens remains a critical bottleneck. Though recent token reduction strategies attempt to accelerate inference, such methods inadequately exploit attention values and fail to address token redundancy. More critically, they overlook the ``attention shift'' phenomenon inherent in LVLMs, which skews token attention scores. In this work, we propose ASAP, a novel training-free, KV-Cache-compatible pruning recipe that comprehensively addresses these limitations. First, we mitigate the attention shift by utilizing a dynamic bidirectional soft attention mask, ensuring the selection of genuinely informative tokens rather than naive attention-based selection. Second, we posit that high semantic redundancy within the token set degrades performance. We therefore introduce a weighted soft merging component that merges semantically similar tokens, preserving only the most feature-dense visual patches for subsequent layers. ASAP achieves virtually lossless compression of visual context, retaining 99.02% of the original LLaVA-NeXT-7B performance while aggressively slashing computational FLOPs by ~80%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。