MadaKV动态优化多模态长文本推理的缓存淘汰，提升效率且不损失精度。
MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference
- 根据注意力头中不同模态偏好自适应调整缓存淘汰策略
- 在长上下文任务中降低1.3至1.5倍推理延迟，内存占用显著减少
- 适合需要高效处理图像+文本等多模态长序列的模型应用
本文提出MadaKV，一种面向多模态大语言模型（MLLMs）长上下文推理的模态自适应键值（KV）缓存淘汰策略。在多模态场景中，不同注意力头对模态的偏好存在差异，导致各注意力头间模态重要性分布不均。传统缓存淘汰方法基于单模态设计，难以捕捉模态特异性信息，性能受限。MadaKV通过模态偏好自适应与分层压缩补偿两大机制，动态感知注意力头内的模态信息并自适应保留关键令牌，在多个代表性MLLM和MileBench基准上的实验表明，其可实现1.3至1.5倍的推理延迟降低，同时保持高精度，显著优于现有缓存淘汰方法。
原文摘要 · Abstract (English)
This paper introduces MadaKV, a modality-adaptive key-value (KV) cache eviction strategy designed to enhance the efficiency of multimodal large language models (MLLMs) in long-context inference. In multimodal scenarios, attention heads exhibit varying preferences for different modalities, resulting in significant disparities in modality importance across attention heads. Traditional KV cache eviction methods, which are tailored for unimodal settings, fail to capture modality-specific information, thereby yielding suboptimal performance. MadaKV addresses these challenges through two key components: modality preference adaptation and hierarchical compression compensation. By dynamically sensing modality information within attention heads and adaptively retaining critical tokens, MadaKV achieves substantial reductions in KV cache memory footprint and model inference decoding latency (1.3 to 1.5 times improvement) while maintaining high accuracy across various multimodal long-context tasks. Extensive experiments on representative MLLMs and the MileBench benchmark demonstrate the effectiveness of MadaKV compared to existing KV cache eviction methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。