通过分离模态内与跨模态注意力，实现更精准的视觉语言推理缓存剪枝。
Cross-Self KV Cache Pruning for Efficient Vision-Language Inference
- 将注意力分解为同模态与跨模态两类，独立处理以提升剪枝精度。
- 在29个数据集上实现最高41%性能提升，缓存占用减少13.6%。
- 无需训练，适合部署在资源受限的多模态生成系统中。
KV缓存剪枝已成为降低长上下文自回归生成内存与计算开销的有前景技术。现有视觉语言模型（VLMs）方法通常依赖大语言模型（LLMs）的自注意力得分来识别并剪除无关标记，但忽略了模态间的分布差异，常导致标记重要性估计不准，过度剪除关键视觉信息。为此，我们提出将注意力得分分解为模态内注意力（同一模态内部）与跨模态注意力（不同模态之间），通过独立管理这两类注意力实现更精确的KV缓存剪枝。此外，引入n-softmax函数以缓解剪枝引发的分布偏移，保持注意力得分原有的平滑性，确保性能稳定。我们的最终训练无关方法——交叉自剪枝（CSP），在与全缓存模型性能相当的同时，显著优于以往剪枝方法。在涵盖29个多模态数据集的MileBench基准上的大量评估表明，CSP在对话式具身对话等挑战性任务上最高实现41%的性能提升，同时将KV缓存预算降低13.6%。代码已公开于https://github.com/TerryPei/CSP。
原文摘要 · Abstract (English)
KV cache pruning has emerged as a promising technique for reducing memory and computation costs in long-context auto-regressive generation. Existing methods for vision-language models (VLMs) typically rely on self-attention scores from large language models (LLMs) to identify and prune irrelevant tokens. However, these approaches overlook the inherent distributional discrepancies between modalities, often leading to inaccurate token importance estimation and the over-pruning of critical visual tokens. To address this, we propose decomposing attention scores into intra-modality attention (within the same modality) and inter-modality attention (across modalities), enabling more precise KV cache pruning by independently managing these distinct attention types. Additionally, we introduce an n-softmax function to counteract distribution shifts caused by pruning, preserving the original smoothness of attention scores and ensuring stable performance. Our final training-free method, \textbf{C}ross-\textbf{S}elf \textbf{P}runing (CSP), achieves competitive performance compared to models with full KV caches while significantly outperforming previous pruning methods. Extensive evaluations on MileBench, a benchmark encompassing 29 multimodal datasets, demonstrate CSP's effectiveness, achieving up to a 41\% performance improvement on challenging tasks like conversational embodied dialogue while reducing the KV cache budget by 13.6\%. The code is available at https://github.com/TerryPei/CSP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。