动态分配多模态长文本缓存，显著提升推理效率。
MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference
- 根据跨模态注意力熵动态调整每层缓存大小。
- 实现72%缓存内存减少，解码速度提升2.82倍。
- 适合长图文、长视频等多模态长上下文任务。
包含长文本-图像和文本-视频模态的多模态大语言模型在输入长度增加时，其多模态键值（KV）缓存规模急剧上升，严重制约推理效率。现有方法在文本与多模态大模型中均忽略各层间注意力密度差异，普遍采用均匀或渐进式压缩策略。本文提出MEDA，一种用于高效多模态长上下文推理的动态分层KV缓存分配方法。核心思想是利用跨模态注意力熵确定每层的KV缓存大小。在动态分配基础上，MEDA采用缓存对选择机制筛选关键KV对，并通过合并策略融合选中与未选中的缓存对，以保留完整上下文信息。实验表明，MEDA在多种多模态长上下文任务中实现最高72%的缓存内存缩减和2.82倍的解码加速，同时保持甚至提升性能，涵盖多图像与长视频场景。代码已开源：https://github.com/AIoT-MLSys-Lab/MEDA。
原文摘要 · Abstract (English)
Long-context Multimodal Large Language Models (MLLMs) that incorporate long text-image and text-video modalities, demand substantial resources as their multimodal Key-Value (KV) caches grow with increasing input lengths, challenging inference efficiency. Existing methods for KV cache compression, in both text-only and multimodal LLMs, have neglected attention density variations across layers, thus often adopting uniform or progressive reduction strategies for layer-wise cache allocation. In this work, we propose MEDA, a dynamic layer-wise KV cache allocation method for efficient multimodal long-context inference. As its core, MEDA utilizes cross-modal attention entropy to determine the KV cache size at each MLLMs layer. Given the dynamically allocated KV cache size at each layer, MEDA also employs a KV pair selection scheme to identify which KV pairs to select and a KV pair merging strategy that merges the selected and non-selected ones to preserve information from the entire context. MEDA achieves up to 72% KV cache memory reduction and 2.82 times faster decoding speed, while maintaining or enhancing performance on various multimodal tasks in long-context settings, including multi-images and long-video scenarios. Our code is released at https://github.com/AIoT-MLSys-Lab/MEDA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。