用状态空间模型动态保留视觉关键信息，提升多模态大模型推理效率
RetentiveKV: State-Space Memory for Uncertainty-Aware Multimodal KV Cache Eviction
- 将键值缓存淘汰从离散删除转为连续记忆演化，基于熵量化低注意力项潜力
- 实现5倍缓存压缩与1.5倍解码加速，有效避免重要视觉特征过早丢弃
- 适合需要长视觉上下文处理的多模态模型优化，尤其关注不确定性感知场景
多模态大语言模型在处理长视觉上下文时面临计算效率与内存消耗的严峻挑战，主要源于视觉键值缓存的显著膨胀。现有缓存压缩方法通常依赖“重要性持久性”假设进行令牌裁剪，但在多模态场景中表现脆弱：1）视觉令牌呈现‘延迟重要性’，初始显著性低但后期解码中变得关键，易被提前淘汰；2）离散裁剪破坏视觉线索的固有空间连续性。为此，我们提出RetentiveKV，一种基于信息熵驱动的键值缓存优化方法，将缓存淘汰从‘离散上下文截断’重构为‘连续记忆演化’，利用状态空间模型（State Space Models）实现。该方法通过熵引导的状态转移，将待淘汰令牌融入连续状态空间，使其在后续解码中语义相关性重现时可动态激活。大规模多模态基准测试表明，RetentiveKV实现了5.0倍的缓存压缩和1.5倍的解码加速。
原文摘要 · Abstract (English)
Multimodal Large Language Models face severe challenges in computational efficiency and memory consumption due to the substantial expansion of the visual KV cache when processing long visual contexts. Existing KV cache compression methods typically rely on the "persistence of importance" hypothesis to prune tokens. However, this approach proves fragile in multimodal settings due to two key issues: 1) Visual tokens display "deferred importance," initially exhibiting low salience but becoming pivotal during later decoding, which can lead to premature eviction. 2) Discrete pruning disrupts the inherent spatial continuity of visual cues. To address these challenges, we propose RetentiveKV, an entropy-driven KV cache optimization method that reformulates KV eviction from "discrete context truncation" to "continuous memory evolution" based on State Space Models. Our method leverages information entropy to quantify the information potential of low-attention tokens and integrates tokens scheduled for eviction into a continuous state space through entropy-guided state transitions, enabling their dynamic reactivation when semantic relevance arises during subsequent decoding. Extensive experiments on multimodal benchmarks demonstrate that RetentiveKV achieves 5.0 times KV cache compression and 1.5 times decoding acceleration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。