VisCache通过分层剪枝减少视觉大模型推理开销，提升效率同时保持性能。
VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inference
- 分两阶段剪枝：先选关键帧，再按注意力动态剪裁键值缓存。
- 仅保留19%~28%缓存即可实现2.35倍加速，性能损失小。
- 无需训练，适配多模态推理场景，尤其适合长序列视觉任务。
视觉大语言模型（VLLM）在多模态推理中表现优异，但其长上下文推理因视觉键值（KV）缓存带来的巨大计算与内存开销而代价高昂。现有KV压缩方法通常对所有视觉标记和层采用统一剪枝，导致信息大量丢失、性能下降。为此，我们提出无需训练的即插即用框架VisCache，采用由粗到细的视觉缓存剪枝策略，包含两个协同阶段：首先，轻量级VLM通过选择性传递语义信息丰富的关键帧来消除时间冗余；其次，我们设计了针对VLLM注意力机制的剪枝算法PruneKV。不同于固定剪枝策略，PruneKV采用抛物线型层间预算分配，并结合非对称更新机制，选择性剪除键（keys）而融合值（values），有效保留关键上下文信息。大量实验表明，VisCache显著提升推理效率，最高达2.35倍加速，内存大幅降低，且仅需保留19%~28%的KV缓存即保持竞争力性能。该方法持续优于现有基线，为长序列VLLM推理建立了新的效率-性能权衡前沿。代码已开源。
原文摘要 · Abstract (English)
While Vision Large Language Models (VLLMs) have achieved remarkable success in multimodal reasoning, their long-context inference remains prohibitively expensive due to the massive computation and memory overhead of visual Key-Value (KV) caches. Existing KV compression methods often apply uniform pruning across visual tokens and layers, leading to substantial information loss and degraded performance.To address this challenge, we propose \textbf{VisCache}, a plug-and-play framework for coarse-to-fine \textbf{Vis}ual KV \textbf{Cache} pruning without training, which consists of two synergistic stages. First, a lightweight VLM filters temporal redundancy by selectively forwarding semantically informative keyframes. Second, we introduce {PruneKV}, a surgical KV compression algorithm tailored to the attention dynamics of VLLMs. Unlike rigid pruning strategies, PruneKV adopts a parabolic layer-wise budget allocation together with an asymmetric update mechanism that selectively prunes keys while fusing values, thereby preserving critical contextual information. Extensive experiments demonstrate that VisCache substantially improves inference efficiency, achieving up to {2.35$\times$ speedup} and significant memory reduction while maintaining competitive performance with only {19--28\%} KV cache retention. VisCache consistently outperforms existing baselines, establishing a new Pareto frontier between efficiency and performance for long-context VLLM inference. Code is available at https://github.com/Wlklk/VisCache
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。