针对多模态模型的缓存管理难题，提出分层自适应淘汰机制。
Hierarchical Adaptive Eviction for KV Cache Management in Multimodal Language Models
- 按视觉与文本注意力差异分层动态淘汰缓存
- 减少41%内存占用，图像理解精度仅降0.3%
- 适合需要高效推理的多模态生成场景
将视觉信息融入大语言模型催生了多模态大模型（MLLMs），但Transformer架构带来的二次方内存与计算开销仍是瓶颈。现有KV缓存淘汰策略未能考虑视觉与文本标记间异构注意力分布，导致效率低下或性能下降。本文提出分层自适应淘汰（HAE）框架，在预填充阶段采用双注意力剪枝（利用视觉标记稀疏性与注意力方差），在解码阶段引入动态淘汰策略（受操作系统回收桶启发）。HAE在多层中最小化KV缓存使用，通过索引广播降低计算开销，并理论证明其信息完整性更优、误差界更低，优于贪心策略。实验表明，该方法在图像理解任务中实现41%的缓存内存压缩，精度仅下降0.3%，且在Phi3.5-Vision-Instruct模型上使故事生成推理速度提升1.5倍，同时保持输出质量。
原文摘要 · Abstract (English)
The integration of visual information into Large Language Models (LLMs) has enabled Multimodal LLMs (MLLMs), but the quadratic memory and computational costs of Transformer architectures remain a bottleneck. Existing KV cache eviction strategies fail to address the heterogeneous attention distributions between visual and text tokens, leading to suboptimal efficiency or degraded performance. In this paper, we propose Hierarchical Adaptive Eviction (HAE), a KV cache eviction framework that optimizes text-visual token interaction in MLLMs by implementing Dual-Attention Pruning during pre-filling (leveraging visual token sparsity and attention variance) and a Dynamic Decoding Eviction Strategy (inspired by OS Recycle Bins) during decoding. HAE minimizes KV cache usage across layers, reduces computational overhead via index broadcasting, and theoretically ensures superior information integrity and lower error bounds compared to greedy strategies, enhancing efficiency in both comprehension and generation tasks. Empirically, HAE reduces KV-Cache memory by 41\% with minimal accuracy loss (0.3\% drop) in image understanding tasks and accelerates story generation inference by 1.5x while maintaining output quality on Phi3.5-Vision-Instruct model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。