通过智能跳过冗余视觉标记，显著提升视觉语言模型训练与推理效率。
Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skipping
- 训练时跳过低效的视觉标记前馈计算，减少冗余开销。
- 推理时动态清除跳过的键值对，降低75%计算量和45%延迟。
- 适合需要高效部署高分辨率多模态大模型的研究与工程团队。
基于Transformer的多模态大语言模型（MLLM）推动了跨模态理解的发展，但随着图像分辨率、训练数据量和模型参数规模的增加，其计算成本急剧上升。主要瓶颈在于细粒度图像理解所需的大量视觉标记。本文提出Skip-Vision，一种统一的框架，解决视觉语言模型在训练和推理中的效率问题。在传统标记压缩基础上，引入两种互补加速策略：训练阶段，发现视觉标记在前馈网络（FFN）中的计算带来边际特征更新，提出跳过（Skip-FFN）策略以规避冗余计算；推理阶段，设计选择性键值缓存移除机制，在解码过程中删除跳过的键值对，同时保持模型性能。实验表明，该方法可将训练时间缩短35%，推理浮点运算量减少75%，延迟降低45%，且性能与现有方法相当或更优。本工作为高效扩展高性能多模态大模型提供了实用方案。
原文摘要 · Abstract (English)
Transformer-based models have driven significant advancements in Multimodal Large Language Models (MLLMs), yet their computational costs surge drastically when scaling resolution, training data, and model parameters. A key bottleneck stems from the proliferation of visual tokens required for fine-grained image understanding. We propose Skip-Vision, a unified framework addressing both training and inference inefficiencies in vision-language models. On top of conventional token compression approaches, our method introduces two complementary acceleration strategies. For training acceleration, we observe that Feed-Forward Network (FFN) computations on visual tokens induce marginal feature updates. This motivates our Skip-FFN strategy, which bypasses FFN layers for redundant visual tokens. For inference acceleration, we design a selective KV-cache removal mechanism that prunes the skipped key-value pairs during decoding while preserving model performance. Experimental results demonstrate that Skip-Vision reduces training time by up to 35\%, inference FLOPs by 75\%, and latency by 45\%, while achieving comparable or superior performance to existing methods. Our work provides a practical solution for scaling high-performance MLLMs with enhanced efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。