发现视觉理解仅依赖少数注意力头，据此实现高效推理优化。
SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs
- 通过响应分析定位视觉相关注意力头，无需额外训练。
- 推理加速1.38倍，内存减少52%，性能几乎无损。
- 适合追求多模态模型推理效率的开发者与研究者。
多模态大语言模型（MLLMs）通常通过扩展预训练语言模型（LLMs）获得视觉能力。本文通过分析注意力机制，揭示了一个意外现象：仅有约5%的注意力头对视觉理解起关键作用，称为视觉头。为高效识别这些头，我们设计了一种无需训练的框架，通过针对性响应分析量化头级别的视觉相关性。基于此发现，提出SparseMM，一种基于视觉评分分配不均衡计算预算的KV缓存优化策略，利用视觉头的稀疏性加速MLLM推理。相比忽略视觉特性的传统方法，SparseMM在解码过程中更注重保留视觉语义。在主流多模态基准上的大量实验表明，SparseMM实现了更优的精度-效率平衡。显著地，在生成阶段实现1.38倍实时加速和52%内存减少，同时保持性能一致。项目已开源：https://github.com/CR400AF-A/SparseMM。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) are commonly derived by extending pre-trained Large Language Models (LLMs) with visual capabilities. In this work, we investigate how MLLMs process visual inputs by analyzing their attention mechanisms. We reveal a surprising sparsity phenomenon: only a small subset (approximately less than 5%) of attention heads in LLMs actively contribute to visual understanding, termed visual heads. To identify these heads efficiently, we design a training-free framework that quantifies head-level visual relevance through targeted response analysis. Building on this discovery, we introduce SparseMM, a KV-Cache optimization strategy that allocates asymmetric computation budgets to heads in LLMs based on their visual scores, leveraging the sparity of visual heads for accelerating the inference of MLLMs. Compared with prior KV-Cache acceleration methods that ignore the particularity of visual, SparseMM prioritizes stress and retaining visual semantics during decoding. Extensive evaluations across mainstream multimodal benchmarks demonstrate that SparseMM achieves superior accuracy-efficiency trade-offs. Notably, SparseMM delivers 1.38x real-time acceleration and 52% memory reduction during generation while maintaining performance parity on efficiency test. Our project is open sourced at https://github.com/CR400AF-A/SparseMM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。