通过压缩视觉令牌，让大模型推理更省显存
Make Your LVLM KV Cache More Lightweight
- 用文本提示引导跨模态信息聚合，逐步压缩视觉特征
- 仅用55%视觉令牌，显存减半、计算降40%
- 适合需要高效视觉语言模型部署的开发者
键值缓存已成为现代大型视觉语言模型（LVLM）推理中的标准组件。尽管它提升了大语言模型的解码效率，但在LVLM中直接使用会因预填充阶段处理大量视觉令牌而带来显著的GPU显存开销。为此，我们提出LightKV，一种新方法，通过挖掘视觉令牌嵌入间的冗余性来减少KV缓存大小。在文本提示引导下，LightKV采用跨模态消息传递机制，在预填充阶段对视觉令牌进行信息聚合与渐进式压缩，其提示感知特性区别于以往仅针对视觉内容的压缩策略。我们在八个开源LVLM上，基于八组公开基准数据集（如MME和SeedBench）进行了评估。结果表明，仅保留原视觉令牌的55%时，LightKV（a）使视觉令牌的KV缓存规模减半，（b）计算量最多降低40%，（c）保持通用性能，同时显著优于现有基线方法。
原文摘要 · Abstract (English)
Key-Value (KV) cache has become a de facto component of modern Large Vision-Language Models (LVLMs) for inference. While it enhances decoding efficiency in Large Language Models (LLMs), its direct adoption in LVLMs introduces substantial GPU memory overhead due to the large number of vision tokens processed during the prefill stage. To tackle this problem, we propose LightKV, a novel approach that reduces KV cache size by exploiting the redundancy among vision-token embeddings. Guided by text prompts, LightKV employs cross-modality message passing to aggregate informative messages across vision tokens and progressively compress them during prefill. This prompt-aware guidance distinguishes our method from prior vision-only compression strategies. We evaluate LightKV on eight open-source LVLMs across eight public benchmark datasets, e.g., MME and SeedBench. Experimental results demonstrate that with only 55% of the original vision tokens, LightKV (a) halves the vision-token KV cache size, (b) reduces computation by up to 40%, and (c) preserves general-purpose performance while significantly outperforming existing baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。