发现多模态大模型处理信息的三阶段规律，实现高效剪枝。
$\mathcal{V}isi\mathcal{P}runer$: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMs
- 基于三阶段信息处理机制，无训练剪枝视觉注意力
- 减少99%视觉注意力计算，降低53.9%计算量
- 适用于多种多模态模型，指导高效架构设计
多模态大语言模型（MLLMs）在视觉-语言任务中表现优异，但因注意力计算随多模态标记数量呈二次增长而带来显著计算开销。尽管已有工作尝试对标记进行剪枝，但缺乏对MLLM如何处理与融合多模态信息的根本理解。通过系统分析，我们揭示了三个阶段的跨模态交互过程：(1) 浅层识别任务意图，视觉标记作为被动注意力汇聚点；(2) 中层发生突变式跨模态融合，由少数关键视觉标记驱动；(3) 深层丢弃视觉标记，专注语言精炼。基于此，我们提出无需训练的剪枝框架VisiPruner，可在LLaVA-v1.5 7B上减少高达99%的视觉相关注意力计算和53.9%的浮点运算量。该方法显著优于现有剪枝方法，并具备跨多种MLLM的泛化能力。此外，我们的发现为训练高效多模态模型提供了可操作的指导，使模型架构与其内在层间处理动态对齐。代码已开源：https://github.com/EIT-NLP/VisiPruner。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have achieved strong performance across vision-language tasks, but suffer from significant computational overhead due to the quadratic growth of attention computations with the number of multimodal tokens. Though efforts have been made to prune tokens in MLLMs, \textit{they lack a fundamental understanding of how MLLMs process and fuse multimodal information.} Through systematic analysis, we uncover a \textbf{three-stage} cross-modal interaction process: (1) Shallow layers recognize task intent, with visual tokens acting as passive attention sinks; (2) Cross-modal fusion occurs abruptly in middle layers, driven by a few critical visual tokens; (3) Deep layers discard vision tokens, focusing solely on linguistic refinement. Based on these findings, we propose \emph{VisiPruner}, a training-free pruning framework that reduces up to 99\% of vision-related attention computations and 53.9\% of FLOPs on LLaVA-v1.5 7B. It significantly outperforms existing token pruning methods and generalizes across diverse MLLMs. Beyond pruning, our insights further provide actionable guidelines for training efficient MLLMs by aligning model architecture with its intrinsic layer-wise processing dynamics. Our code is available at: https://github.com/EIT-NLP/VisiPruner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。