提出分阶段剪枝方法，提升视觉令牌多样性与任务相关性。
When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics
- 分两阶段剪枝：先用排斥机制保证空间结构多样性，再用指令感知注意力剔除无关令牌。
- 相比传统方法，保留的视觉令牌在结构上更分散，语义上更贴合任务需求。
- 适合追求高效且精准的视觉语言模型推理优化的研究者与开发者。
视觉语言模型虽能力强大，但推理时计算开销大。现有视觉令牌剪枝方法主要依赖初始注意力分数，但高注意力值会自然聚集于语义相似区域，导致特征多样性严重下降，丢失关键上下文信息。为此，本文提出结构到语义（STS）的两阶段视觉令牌剪枝框架，显式解耦剪枝过程：第一阶段采用基于排斥的采样机制，最大化空间与结构多样性；第二阶段利用指令感知交叉注意力，精确过滤出与提示无关的令牌。两阶段协同作用，先保障几何覆盖，再按语义相关性优化保留令牌。大量实验表明，STS有效缓解了基于注意力选择带来的冗余问题，提升了保留视觉令牌的结构多样性和细粒度任务对齐能力。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have demonstrated remarkable capabilities but suffer from significant computational overhead during inference. While visual token pruning offers a promising solution, existing methods predominantly rely on initial attention scores. This single-metric paradigm presents a critical flaw: high attention scores inherently collapse onto semantically similar regions, thereby severely reducing feature diversity and discarding vital contextual details. To address this, we introduce Structure-to-Semantics (STS), a novel two-stage visual token pruning framework that explicitly decouples the pruning process. The first stage employs a repulsion-based sampling mechanism to maximize spatial and structural diversity. The second stage leverages instruction-aware cross-attention to precisely filter out prompt-irrelevant tokens. This two-stage synergy constitutes the core of STS, first ensuring geometric coverage and then refining the retained tokens according to semantic relevance. Extensive evaluations demonstrate that STS mitigates the redundancy caused by attention-based selection, improving both structural diversity and fine-grained task alignment of the preserved visual tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。