通过剪枝冗余视觉特征，显著降低视觉语言模型推理开销。
Efficient LLaMA-3.2-Vision by Trimming Cross-attended Visual Features
- 利用跨注意力图稀疏性，选择性删除冗余视觉特征
- 减少50%视觉特征后，延迟与内存使用下降，性能不变
- 无需额外训练，适合部署在资源受限场景
视觉标记压缩可降低大型视觉语言模型（LVLMs）中大量图像特征带来的推理成本。与以往仅针对自注意力模型的研究不同，本文聚焦于性能更优的跨注意力模型。我们发现，跨注意力层中图像标记的键值（KV）缓存大小远超自注意力层中文本标记的缓存，成为主要计算瓶颈。为此，我们利用跨注意力图的稀疏特性，选择性剪枝冗余视觉特征。所提出的剪枝版Llama能有效降低KV缓存需求，且无需额外训练。得益于50%的视觉特征缩减，模型在保持基准性能的同时，显著降低推理延迟和内存占用。
原文摘要 · Abstract (English)
Visual token reduction lowers inference costs caused by extensive image features in large vision-language models (LVLMs). Unlike relevant studies that prune tokens in self-attention-only LVLMs, our work uniquely addresses cross-attention-based models, which achieve superior performance. We identify that the key-value (KV) cache size for image tokens in cross-attention layers significantly exceeds that of text tokens in self-attention layers, posing a major compute bottleneck. To mitigate this issue, we exploit the sparse nature in cross-attention maps to selectively prune redundant visual features. Our Trimmed Llama effectively reduces KV cache demands without requiring additional training. By benefiting from 50%-reduced visual features, our model can reduce inference latency and memory usage while achieving benchmark parity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。