通过冻结无效层的视觉令牌，让多模态大模型更高效
ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layers
- 用新指标衡量每层对图文令牌的影响，找出低效层
- 可冻结约60%层的视觉令牌更新，减少50%计算量
- 无需训练，适合追求推理效率的部署场景
多模态大语言模型因规模庞大和视觉令牌数量多，导致计算成本高昂。本文提出一种新度量方法 Layer Contribution（LC），用于量化每层对视觉与文本令牌的影响。通过移除某层对特定令牌的变换并观察输出偏差来计算。初步实验发现，许多层在处理视觉令牌时贡献极小。基于此，提出 ShortV——一种无需训练的方法，利用 LC 识别无效层，并冻结这些层中视觉令牌的更新。实验表明，ShortV 可在约 60% 的层中冻结视觉令牌更新，显著降低计算开销。例如，在 LLaVA-NeXT-13B 上实现 50% 的 FLOPs 下降，同时保持优异性能。代码将公开于 https://github.com/icip-cas/ShortV
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) suffer from high computational costs due to their massive size and the large number of visual tokens. In this paper, we investigate layer-wise redundancy in MLLMs by introducing a novel metric, Layer Contribution (LC), which quantifies the impact of a layer's transformations on visual and text tokens, respectively. The calculation of LC involves measuring the divergence in model output that results from removing the layer's transformations on the specified tokens. Our pilot experiment reveals that many layers of MLLMs exhibit minimal contribution during the processing of visual tokens. Motivated by this observation, we propose ShortV, a training-free method that leverages LC to identify ineffective layers, and freezes visual token updates in these layers. Experiments show that ShortV can freeze visual token in approximately 60\% of the MLLM layers, thereby dramatically reducing computational costs related to updating visual tokens. For example, it achieves a 50\% reduction in FLOPs on LLaVA-NeXT-13B while maintaining superior performance. The code will be publicly available at https://github.com/icip-cas/ShortV
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。