让视觉语言模型按提示智能删冗余图像片段，提速省显存
ZSPAPrune: Zero-Shot Prompt-Aware Token Pruning for Vision-Language Models
- 基于提示感知机制，优先保留与任务相关的关键图像区域
- 剪枝90%图像令牌仍保持接近顶尖性能，精度损失极小
- 无需训练即可用，适合追求高效推理的部署场景
随着视觉语言模型能力提升，其处理的输入规模日益增大，但相较于大语言模型，视觉输入常产生显著的图像令牌冗余，导致高昂的推理成本。现有剪枝方法多依赖注意力或多样性，却忽视文本提示的引导，难以有效聚焦任务相关区域。本文提出一种全新的零样本方法，将剪枝问题重构为任务相关性与信息多样性之间的权衡。该方法采用分层策略：先选取任务相关的核心图像令牌，再补充多样性令牌以保留更广泛上下文。在多个模型和基准上的实验表明，本方法在剪枝高达90%图像令牌的情况下，性能仍可媲美或超越现有最先进方法，且仅伴随微小精度损失；同时显著降低GPU显存占用与推理延迟。
原文摘要 · Abstract (English)
As the capabilities of Vision-Language Models (VLMs) advance, they can process increasingly large inputs, which, unlike in LLMs, generates significant visual token redundancy and leads to prohibitive inference costs. While many methods aim to reduce these costs by pruning visual tokens, existing approaches, whether based on attention or diversity, typically neglect the guidance of the text prompt and thus fail to prioritize task relevance. In this work, we propose a novel, zero-shot method that reframes the problem by introducing a prompt-aware perspective, explicitly modeling visual token pruning as a balance between task relevance and information diversity. Our hierarchical approach first selects a core set of task-relevant visual tokens and then supplements them with diversity tokens to preserve broader context. Experiments across multiple models and benchmarks show that our method achieves performance that matches or surpasses the state-of-the-art with only minimal accuracy loss, even when pruning up to 90\% of the tokens. Furthermore, these gains are accompanied by significant reductions in GPU memory footprint and inference latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。