Nüwa通过两阶段剪枝保持视觉空间完整性，显著提升模型在定位任务上的表现。
Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning
- 先分离、对齐、聚合保留空间锚点，再用文本引导剪枝保留关键视觉信息
- 在视觉问答任务上达到94%-95%准确率，定位任务提升7%至47%
- 适合需要高精度空间理解的多模态应用，如视觉定位与交互
视觉标记剪枝已被证明是高效视觉语言模型（VLM）的有效加速技术。然而，现有剪枝方法在视觉问答（VQA）任务中表现良好，却在视觉定位（VG）任务中出现显著性能下降。我们分析VLM处理流程发现，基于全局语义相似性和注意力分数的策略会丢失由标记位置信息交互产生的全局空间参考框架。受此启发，我们提出Nüwa，一种两阶段标记剪枝框架，在实现高效特征聚合的同时保持空间完整性。第一阶段在视觉编码器后，引入分离、对齐和聚合三个操作，灵感来自群体智能算法，以保留富含信息的全局空间锚点。第二阶段在大语言模型中进行文本引导剪枝，保留与任务相关的视觉标记。大量实验表明，Nüwa在多个VQA基准上达到当前最优（94%至95%），并在视觉定位任务上取得7%至47%的显著提升。
原文摘要 · Abstract (English)
Vision token pruning has proven to be an effective acceleration technique for the efficient Vision Language Model (VLM). However, existing pruning methods demonstrate excellent performance preservation in visual question answering (VQA) and suffer substantial degradation on visual grounding (VG) tasks. Our analysis of the VLM's processing pipeline reveals that strategies utilizing global semantic similarity and attention scores lose the global spatial reference frame, which is derived from the interactions of tokens' positional information. Motivated by these findings, we propose $\text{Nüwa}$, a two-stage token pruning framework that enables efficient feature aggregation while maintaining spatial integrity. In the first stage, after the vision encoder, we apply three operations, namely separation, alignment, and aggregation, which are inspired by swarm intelligence algorithms to retain information-rich global spatial anchors. In the second stage, within the LLM, we perform text-guided pruning to retain task-relevant visual tokens. Extensive experiments demonstrate that $\text{Nüwa}$ achieves SOTA performance on multiple VQA benchmarks (from 94% to 95%) and yields substantial improvements on visual grounding tasks (from 7% to 47%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。