通过精简视觉令牌加速多模态模型的图像视频分割任务
EVTP-IVS: Effective Visual Token Pruning For Unifying Instruction Visual Segmentation In Multi-Modal Large Language Models
- 基于空间信息优化的令牌筛选策略，保留关键视觉特征
- 视频任务提速5倍，仅用20%令牌仍保持高精度
- 适合需要高效推理的实时视频理解场景
指令式视觉分割（IVS）要求根据自然语言指令对图像或视频中的物体进行分割。尽管近期多模态大模型（MLLMs）在IVS任务上表现强劲，但其推理开销仍是主要瓶颈，尤其在视频处理中更为突出。我们实证分析了MLLMs中的视觉令牌采样，发现令牌子集覆盖度与分割性能呈强相关性。由此启发，我们设计了一种简单有效的视觉令牌剪枝方法，选择紧凑且空间代表性强的令牌子集以加速推理。本文提出一种新的针对IVS的视觉令牌剪枝方法EVTP-IV，基于k-center并融合空间信息，确保更优覆盖。我们进一步提供了信息论分析支持该设计。在标准IVS基准上的实验表明，该方法在视频任务上实现最高5倍加速，在图像任务上达3.5倍加速，同时仅使用20%的令牌即保持相当精度。在不同剪枝比例下，该方法始终优于现有先进剪枝基线。
原文摘要 · Abstract (English)
Instructed Visual Segmentation (IVS) tasks require segmenting objects in images or videos based on natural language instructions. While recent multimodal large language models (MLLMs) have achieved strong performance on IVS, their inference cost remains a major bottleneck, particularly in video. We empirically analyze visual token sampling in MLLMs and observe a strong correlation between subset token coverage and segmentation performance. This motivates our design of a simple and effective token pruning method that selects a compact yet spatially representative subset of tokens to accelerate inference. In this paper, we introduce a novel visual token pruning method for IVS, called EVTP-IV, which builds upon the k-center by integrating spatial information to ensure better coverage. We further provide an information-theoretic analysis to support our design. Experiments on standard IVS benchmarks show that our method achieves up to 5X speed-up on video tasks and 3.5X on image tasks, while maintaining comparable accuracy using only 20% of the tokens. Our method also consistently outperforms state-of-the-art pruning baselines under varying pruning ratios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。