分析视觉令牌剪枝中不同角色令牌的作用差异，发现非活跃令牌也会影响模型表现。
Not All Redundant Tokens Are Alike: Analyzing Visual Token Pruning through Token Roles
- 基于令牌功能角色分析剪枝策略的偏差
- 保留弱语义对齐令牌可维持甚至提升性能
- 适合关注模型推理效率与可解释性的研究者
视觉语言模型（VLMs）将图像处理为视觉令牌序列，导致推理阶段存在显著计算瓶颈。近期的视觉令牌剪枝方法通过移除看似冗余的令牌来缓解这一问题，但这些剪枝决策与视觉令牌的功能角色之间的关系仍不明确。本文通过EmbedLens识别的令牌角色视角分析剪枝行为，发现主流剪枝方法具有不同的角色偏好，但这些偏好与下游任务性能无直接关联。为此，我们改进了角色分配流程，并评估了角色保护型剪枝变体。结果表明，在某些情况下，保留非活跃（non-alive）令牌反而能维持或提升模型性能，说明即使与语义关联较弱的令牌，在剪枝条件下仍可能影响模型行为。代码已开源。
原文摘要 · Abstract (English)
Vision-language models (VLMs) process an image as a sequence of visual tokens, which creates a substantial computational bottleneck during inference. Recent visual token pruning methods address this issue by removing seemingly redundant tokens, yet it remains unclear how these pruning decisions relate to the functional roles of visual tokens. In this work, we analyze visual token pruning through the lens of token roles identified by EmbedLens. We first show that representative pruning methods exhibit distinct token-role biases, but these biases do not directly correlate with downstream performance. To better understand this behavior, we refine the token-role assignment procedure and evaluate role-protected pruning variants. Our results show that preserving non-alive tokens can sometimes maintain or improve performance, suggesting that tokens with weak direct semantic alignment may still affect model behavior under pruning. Our code is publicly available at https://github.com/jaykim9870/Not_All_Redundant_Tokens_Are_Alike.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。