针对多模态大模型设计自适应剪枝方法，提升压缩效率与性能。
TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models
- 按层调整稀疏率，依据多模态输出的多样性动态保留参数。
- 利用注意力分数筛选关键输入令牌，指导非结构化权重剪枝。
- 在视觉语言和音视频任务模型上均显著优于现有剪枝方法。
多模态大语言模型（MLLMs）在理解多种模态数据和任务方面表现出强大能力，但其模型规模也随之增大。尽管训练后剪枝能减少单模态模型体积，但在多模态模型上的应用效果有限。我们分析发现，传统方法未考虑多模态模型中各层和模态间特有的令牌属性。受此启发，我们提出TAMP，一种专为多模态大模型设计的简单而有效的剪枝框架，包含两个核心组件：(1) 多样性感知稀疏性，根据多模态输出令牌间的多样性动态调整每层的稀疏率，高多样性层保留更多参数；(2) 自适应多模态输入激活，通过注意力得分识别代表性多模态输入令牌，指导非结构化权重剪枝。我们在两种先进多模态大模型上验证该方法：面向视觉-语言任务的LLaVA-NeXT和可处理音频、视觉与语言模态的VideoLLaMA2。在多个多模态评估基准上的实验证明，该方法的每个组件均显著优于现有剪枝技术。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have shown remarkable versatility in understanding diverse multimodal data and tasks. However, these capabilities come with an increased model scale. While post-training pruning reduces model size in unimodal models, its application to MLLMs often yields limited success. Our analysis discovers that conventional methods fail to account for the unique token attributes across layers and modalities inherent to MLLMs. Inspired by this observation, we propose TAMP, a simple yet effective pruning framework tailored for MLLMs, featuring two key components: (1) Diversity-Aware Sparsity, which adjusts sparsity ratio per layer based on diversities among multimodal output tokens, preserving more parameters in high-diversity layers; and (2) Adaptive Multimodal Input Activation, which identifies representative multimodal input tokens using attention scores to guide unstructured weight pruning. We validate our method on two state-of-the-art MLLMs: LLaVA-NeXT, designed for vision-language tasks, and VideoLLaMA2, capable of processing audio, visual, and language modalities. Empirical experiments across various multimodal evaluation benchmarks demonstrate that each component of our approach substantially outperforms existing pruning techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。