用视觉语言引导剪枝，让分割模型更快更省资源。
VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation
- 根据任务需求动态筛选图像片段，只处理相关部分。
- 计算量降低25%无性能损失，40%仅降1%精度。
- 适合多模态大模型驱动的复杂分割任务。
视觉变换器（ViTs）已成为众多分割模型的核心，持续实现最先进性能，但其计算开销巨大。图像标记剪枝是缓解此问题的有效策略。然而，现有方法在更复杂的任务导向分割（TOS）中表现不足，因每个图像块的类别并非预定义，而是依赖具体输入任务。本文提出视觉语言引导标记剪枝（VLTP），一种新型剪枝机制，可加速基于ViT的分割模型，尤其适用于由多模态大语言模型（MLLM）指导的TOS。我们主张，ViT无需在所有层处理每个图像标记——仅需关注与推理任务相关的标记。设计新的剪枝解码器，同时输入图像标记和视觉-语言引导信息，以预测每标记对任务的相关性。仅高相关性标记传递至ViT深层。实验表明，该框架使ViT计算成本降低约25%且性能无下降，降低约40%时仅损失1%性能。相关代码可在该网址获取。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) have emerged as the backbone of many segmentation models, consistently achieving state-of-the-art (SOTA) performance. However, their success comes at a significant computational cost. Image token pruning is one of the most effective strategies to address this complexity. However, previous approaches fall short when applied to more complex task-oriented segmentation (TOS), where the class of each image patch is not predefined but dependent on the specific input task. This work introduces the Vision Language Guided Token Pruning (VLTP), a novel token pruning mechanism that can accelerate ViT-based segmentation models, particularly for TOS guided by multi-modal large language model (MLLM). We argue that ViT does not need to process every image token through all of its layers -- only the tokens related to reasoning tasks are necessary. We design a new pruning decoder to take both image tokens and vision-language guidance as input to predict the relevance of each image token to the task. Only image tokens with high relevance are passed to deeper layers of the ViT. Experiments show that the VLTP framework reduces the computational costs of ViT by approximately 25% without performance degradation and by around 40% with only a 1% performance drop. The code associated with this study can be found at this URL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。